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Preface 


Immersive technologies such as augmented reality (AR) and virtual reality (VR) are 
changing the business landscape, providing new opportunities but also concerns for 
businesses and consumers. Organised by the Creative Augmented and Virtual 
Reality Hub at Manchester Metropolitan University, the 4th International 
Augmented and Virtual Reality Conference attracted researchers and industry from 
around the globe to discuss opportunities, collaborations and future research 
directions. The conference theme of “The Power of AR and VR for Business” 
invited academic and industry speakers from various disciplines, to share their 
knowledge and experiences of immersive technologies. 

Papers presented focussed on the areas of retail, tourism, experience design, 
education and applications and immersive designs. We hope that the conference 
and this book will serve as a valuable source for future research and discussion on 
important issues such as privacy, technology adoption and application design. In 
addition, this book aims to inform businesses about latest developments in the areas 
of AR and VR. 


Manchester, UK Dr. M. Claudia tom Dieck 
Dr. Timothy Jung 


International Augmented and Virtual Reality 
Conference 2018 


Scientific Committee 


Patrick Allen, University of Bradford 

Mario Ascencao, Haaga-Helia AUS 

Alexander Brem, Friedrich-Alexander-Universitat Erlangen-Nirnberg 
Namho Chung, Kyung Hee University 

Dario tom Dieck, Manchester Metropolitan University 

M. Claudia tom Dieck, Manchester Metropolitan University 
Peter Eachus, University of Salford 

Alex Gibson, Dublin Institute of Technology 

Dai-In Han, NHTV Breda University of Applied Sciences 
Ana Javornik, Newcastle University 

Sarah Jones, Birmingham City University 

Timothy Jung, Manchester Metropolitan University 

Si Jung Kim, University of Nevada, Las Vegas 

Yen-Soon Kim, University of Nevada, Las Vegas 

Richard Koeck, University of Liverpool 

Nina Krey, Rowan University 

Slimane Larabi, USTHB University 

Cynthia Mejia, University of Central Florida 

Andy Miah, University of Salford 

Eleni Michopoulou, University of Derby 

Hossein Olya, Oxford Brookes University 

Mary O’Rawe, Dublin Institute of Technology 

B. Joseph Pine I, Columbia University 

Lee Quinn, University of Manchester 

Philipp Rauschnabel, Universitat der Bundeswehr Miinchen 
Alexander Rossmann, Reutlingen University 


vili International Augmented and Virtual Reality Conference 2018 


Caroline Scarles, University of Surrey 

Michael Schwertel, Cologne Business School 

Pasi Tuominen, Haaga-Helia University of Applied Sciences 
Gary Warnaby, Manchester Metropolitan University 


Contents 


PartI AR & VR Retail Experience 


Augmented Reality in Real Stores: Empirical Evidence 


from Consumers’ Interaction with AR in a Retail Format.......... 


Francesca Bonetti, Eleonora Pantano, Gary Warnaby, Lee Quinn 
and Patsy Perry 


V-Commerce in Retail: Nature and Potential Impact.............. 


Anouk de Regt and Stuart J. Barnes 


A Virtual Reality and Retailing Literature Review: Current 


Focus, Underlying Themes and Future Directions ................ 


Liangchao Xue, Christopher J. Parker and Helen McCormick 


Part II AR & VR Experience Design 
What We Don’t Know. The Effect of Realism in Virtual Reality 


on Experience and Behaviour......................000 00s eee 


Marnix van Gisbergen, Michelle Kovacs, Fabio Campos, 
Malou van der Heeft and Valerie Vugts 


Adapting Jake Knapp’s Design Sprint Approach for AR/VR 


Applications in Digital Heritage..........................0002. 


Helen Southall, Maeve Marmion and Andrew Davies 


Part II AR & VR in Tourism 
Designing Valuable Augmented Reality Tourism Application 


PROEFIENGOS) shoh ad hwo eevee ba epa aes PER ESERI SE RE TEES wakes 


Eleanor E. Cranmer 


Experiencing Virtual Reality in Heritage Attractions: Perceptions 


OF Fliderly W866 5.6.6 4562406548 ch ed ei dd OARS REnD SER HERDS ORES 


M. Claudia tom Dieck, Timothy Jung and Eleni Michopoulou 


A Case Study: Assessing Effectiveness of the Augmented Reality 


Application in Augusta Raurica.....................00.000505 


Moritz Armingeon, Pleurat Komani, Trupti Zanwar, Safak Korkut 
and Rolf Dornberger 


Virtual and Augmented Reality Technologies to Enhance 


the Visitor Experience in Cultural Tourism................... 


Dai-In Danny Han, Jessika Weber, Marcel Bastiaansen, Ondrej Mitas 
and Xander Lub 


Tourism Marketers Perspectives on Enriching Visitors City 


Experience with Augmented Reality: An Exploratory Study ...... 


Natasha Moorhouse, Timothy Jung and M. Claudia tom Dieck 


PartIV AR & VR in Education 


Creating Virtual Reality in a Business and Technology 


Educational Context...........0.0.00 0000000. ee 


Diana Andone and Mark Frydenberg 
Immersive Virtual Reality IVR) in Higher Education: 


Development and Implementation .......................... 


Paula Hodgson, Vivian W. Y. Lee, Johnson C. S. Chan, Agnes Fong, 
Cindi S. Y. Tang, Leo Chan and Cathy Wong 


Cultural Heritage Objects in Education by Virtual 


and Augmented Reality ............00 0.0.0.0... cece eee eee 


Jan Lacko 


Part V AR & VR Applications and Immersive Designs 
To Have and Vehold: Marrying Museum Objects and Virtual 


Collections via AR .........0 000.0000 cece ee eee 


Ronald Haynes 
A Tool, not a Toy: Using Virtual Reality to Evaluate the 


Communication Between Autonomous Vehicles and Pedestrians . . . 


Sebastian Stadler, Henriette Cornet, Tatiana Novaes Theoto 
and Fritz Frenkler 


Designing Spatial UI as a Solution of the Narrow FOV 


of Microsoft HoloLens: Prototype of Virtual Museum Guide...... 


Ramy Hammady and Minhua Ma 


Recommender System as the Support for Binaural Audio........ 


David Bernhauer and Tomas Skopal 


Contents 


Contents xi 


Virtual Reality References in Design Problem Solving: 

Towards an Understanding of Affect-Cognition Interaction 

in Conceptual Design ............... 00.0.0 eee 247 
R. Vimal Krishnan and Prasad S. Onkar 


Intuitive Hand Gestures for the Interaction with Information 

Visualizations in Virtual Reality........................0000005 261 
Gabriel Frey, Arno Jurkschat, Safak Korkut, Jonas Lutz 

and Rolf Dornberger 


Part VI AR & VR Medical Applications 


Pulmonary Rehabilitation in Virtual Reality for COPD 

PAW 64 ctesdearebewsssd eeemideds bodied hier ascnbegeead 277 
Natasha Moorhouse, Timothy Jung, Xin Shi, Farhan Amin, 

Joanne Newsham and Sarah McCall 


Exploring Surgeon’s Acceptance of Virtual Reality Headset 
Foy TYSONS od ohased sd eciew O64 506Gee ne etegetebs ieee 291 
Libi Beke Hen 


Evaluation of Virtual Reality in Orthopaedic 

Training—A Pioneering Pilot Study.........................0.. 305 
Ronnie Davies, Natasha Moorhouse, Timothy Jung, Saleem Mastan 

and Bibhas Roy 


Part VII VR and Media 


Towards the Essence of Cinematic VR: Embracing 
New Technologies to Define a Medium.......................... 321 
Sarah Jones 


Part I 
AR & VR Retail Experience 


Augmented Reality in Real Stores: Mm) 
Empirical Evidence from Consumers’ sche 
Interaction with AR in a Retail Format 


Francesca Bonetti, Eleonora Pantano, Gary Warnaby, Lee Quinn 
and Patsy Perry 


Abstract This exploratory empirical study elucidates the concept of the 
‘augmented store’, namely a physical retail store modified to accommodate AR 
technology. It extends previous research into immersive environments and 
technology-enhanced stores from experimental laboratory settings to a real-life 
scenario with participating consumers. Qualitative data from interviews and 
observations of consumers using AR technology in-store are analysed to evidence 
naturalistic understandings of interactions with, and perceptions of, a physical store 
enhanced with AR technologies. The findings provide evidence to suggest that 
consumers experience an enhanced, more immersive and enjoyable perception of 
the store environment as a consequence of the AR experience. They find interaction 
with the augmented store to be ‘realistic’, and hedonic motivations for interacting 
with the immersive store frequently prevail. The AR enhanced store appears to 
stimulate brand engagement, increasing consumers’ desire to shop at the retailer, 
which provides managerial opportunities to reinforce brand positioning. 


Keywords Retailing - Human-computer interaction - Augmented store - 
Augmented reality - Consumer behaviour - Immersion 
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1 Introduction 


Atmospherics are acknowledged as critical elements of the retail store environment, 
which help convey a satisfactory shopping experience and, ultimately, influence 
consumer behaviour (Kotler, 1973; Puccinelli et al., 2009). Such factors are 
arguably even more relevant in fashion and apparel retailing, given the 
multi-sensory experience of both store environment and the products themselves 
(Foster & McLelland, 2014). In a constantly evolving competitive context, fashion 
and apparel retailers are increasingly adopting different types of innovative tech- 
nologies in physical stores to directly enhance the shopping experience, and thereby 
achieve competitive advantage (Pantano, 2016). Such technologies (see Bonetti & 
Perry, 2017; McCormick et al., 2014; Pantano, Rese, & Baier, 2017) can be referred 
to as ‘consumer-facing’ in-store technology (Bonetti & Perry, 2017). 

‘Immersive’ technologies (e.g. AR and VR) are rapidly evolving and are 
increasingly adopted in this context (Bonetti, Warnaby, & Quinn, 2017; Javornik, 
2016). In particular, AR applications are developing to combine both the real and 
virtual worlds into the user’s view of the physical world in real time (Carmigniani 
et al., 2011). This can help enhance the user’s visualisation of products and per- 
ception of the store environment, and thus, the shopping experience, by enabling 
interaction with virtual items (Huang & Liao, 2015). Whilst VR blocks out real 
world sensory experiences through a wearable device (typically a headset), 
immersing the user in virtual and entertaining 3D worlds (Bonetti et al., 2017), AR 
allows users to experience enhanced and more realistic experiences within the 
physical place (Papagiannidis, Pantano, See-To, Dennis, & Bourlakis, 2017). 

AR is defined as a combination of ‘real and computer-generated digital infor- 
mation into the user’s view of the physical world in such a way they appear as 
one environment’ (Olsson, Lagerstam, Karkkainen, & Vaananen, 2013, p. 288). By 
integrating and aligning real and virtual objects (through a virtual layer that can add 
computer-generated digital elements such as images, videos, textual information, 
etc.), this technology results in an enhanced (augmented) physical world 
(Carmigniani et al., 2011; Pantano et al., 2017). Although existing studies have 
investigated immersive environments and technology-enhanced stores, this research 
has tended to have been conducted in experimental settings (Huang & Liao, 2015; 
Kjeldskov & Graham, 2003; Papagiannidis et al., 2017). Therefore, the integration 
of immersive AR tools within the traditional point-of-sale store environment (i.e. 
stores that already exist, which are modified to accommodate AR technology, rather 
than having AR features built into the initial store concept) to investigate users’ 
interaction with—and perception of—stores enhanced with immersive AR tech- 
nologies in a real store environment is still under-investigated. This leads to our 
research questions: 


RQ1. To what extent can traditional stores integrate immersive technologies such as 
AR, to develop new store forms/concepts? 

RQ2. How do consumers perceive the store environment of a more traditional, 
physical store which has been enhanced by AR technologies? 
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RQ3. How do consumers interact with a traditional store enhanced with AR 
technologies? 


Using an apparel store enhanced with immersive technologies as a research 
context, the central aim of this study is to investigate the extent to which immersive 
AR technologies influence the way consumers interact with, perceive and respond 
to retail settings and store environments. 


2 Theoretical Background 


2.1 Augmented Reality in Retailing 


A major theme in the existing literature relates to the way(s) in which users adopt, 
interact with, and experience technology devices and systems (see Dix, 2009; 
Kjeldskov & Graham, 2003; Rogers, 2004). Of the various forms of innovative 
technologies used in retail environments, immersive technologies have drawn 
particular attention (Bonetti et al., 2017; Javornik, 2016; Pantano et al., 2017), 
particularly AR, which is based on a camera able to capture real-world data and 
combine information from real and virtual sources into one perception (Oleksy & 
Wnuk, 2016). Consequently, product simulation, sound, GPS data and media 
richness contribute to experiential value, with AR enabling consumers to interact 
with virtual products (McCormick et al., 2014). AR applications have become more 
popular due to widely distributed personal mobile technology, allowing users to 
shop using AR, thereby enhancing satisfaction and experience (Dacko, 2016; 
Javornik, 2016). Early AR retail applications include virtual try-on, and interactive 
displays providing information on promotion, products and locations (Bonetti et al., 
2017). Thus, AR has the potential to improve consumers’ visualisation of products, 
increase engagement and enhance perceptions of the shopping experience, thereby 
hopefully affecting retailer and brand perception positively which, in turn, can 
influence consumer behaviour (Huang & Liao, 2015; McCormick et al., 2014). 


2.2 Augmented Places 


The development of new immersive technologies contributes to the creation of 
immersive (augmented) places/environments, increasing users’ levels of engage- 
ment, enjoyment and satisfaction, leading to the enhancement of the user experience 
(Papagiannidis et al., 2017). Augmented places consist of real, physical places, 
enhanced by AR technologies to augment users’ overall current perception of reality, 
their experience, and the possibilities offered by the real world (Carmigniani et al., 
2011; Oleksy & Wnuk, 2016; Pantano et al., 2017). This can lead to a deeper level of 
engagement, enjoyment and satisfaction (Dacko, 2016; Papagiannidis et al., 2017). 
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Existing literature on immersive places arising from the use of AR technology 
mainly focuses on the entertainment and educational sectors, and museums/other 
places of historic cultural heritage (Chang, Hou, Pan, Sung, & Chang, 2015). AR 
technologies potentially enable a deeper place-based participation by allowing users 
to virtually, yet naturally, experience an enhanced version of the physical space in 
real-time via realistic interfaces. This increases feelings of immersion and engage- 
ment (Oleksy & Wnuk, 2016; Pantano et al., 2017), and can result in improved 
perceptions of the experience and the real environment, by offering new possibilities 
to see objects not physically available in the real-world context, enriching content 
and information at users’ disposal. This creates an immersive environment and 
interesting, fun experiences (Oleksy & Wnuk, 2016; Papagiannidis et al., 2017). 
Oleksy and Wnuk (2016) posited the concept of ‘augmented places’ as physical 
places enhanced with AR with the aim to recreate and enhance the experience of the 
place in question, by overlapping virtual reconstructions of past heritage and actual 
place, supporting users’ understanding of its historical value, and resulting in users’ 
higher emotional attitude towards the place and greater understanding of meaning of 
multicultural places (Oleksy & Wnuk, 2016). These elements of immersive envi- 
ronments and augmented places could potentially be extended to the retail context. 


2.3. Human-Computer Interaction (HCI) in Retail Settings 


In order to facilitate technology acceptance, developers need to provide interac- 
tional modality that is as realistic and natural as possible (Carmigniani et al., 2011). 
Consumers’ acceptance of, and interaction with, technological innovations in retail 
settings has received greater attention, due to the growing adoption of technologies 
at the point of sale to enhance customer experience and increase competitiveness 
(Bonetti & Perry, 2017; Pantano & Gandini, 2017). 

Indeed, research into human-computer interaction (HCI) in retail settings enri- 
ched with enhanced and immersive AR and VR technologies has expanded in 
recent years. Conducted in a laboratory environment, Pantano et al.’s (2017) study 
investigated the effect of customer interaction with AR technologies when trying on 
glasses to simulate virtual fit and appearance. Their results showed aesthetic quality 
and interactivity to be antecedents of perceived ease of use; response time and 
quality of information influenced consumers’ positive attitude; and when combined 
with the perceived enjoyment in interacting with the technology, improved the 
online buying decision process. Olsson et al.’s (2013) assessment of potential end 
users’ expectations and requirements of future mobile AR services characteristics 
and user experience in a shopping centre context revealed that participants expect 
the technology to be proactive and context-aware, suggesting products and activ- 
ities based on the user’s location, as well as providing relevant and personalised 
content, with interaction intuitive, natural and easy to learn, flexible and controlled 
by the user. Dacko’s (2016) examination of mobile AR apps and the extent to 
which they contribute to smart retail settings found user satisfaction to be relatively 
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high, and that technology use provides experiential shopping benefits, including 
more efficient or better value shopping, more entertaining and more visually 
appealing shopping. 

Research on AR in a retail context mainly concerns online stores, conducted in 
controlled laboratory environments (Huang & Liao, 2015; Pantano & Laria, 2012; 
Papagiannidis et al., 2017). However, as physical stores have adopted these tech- 
nologies, there is an increasing need and opportunity to conduct research in this 
particular real-world context. In their review of research methods applied within 
HCI for mobile devices, Kjeldskov and Graham (2003) noted the tendency towards 
building systems and evaluating consumers’ usage and interaction within artificially 
controlled environments and isolated laboratory-based settings, at the expense of 
understanding and learning from the actual use of technologies in ‘messy’ 
real-world contexts (Dix, 2009), characterised by distractions, noise and interrup- 
tions (Rogers, 2004). 


3. Methodology 


Qualitative research inquiry was used to evaluate users’ reaction to immersion in an 
AR-enhanced store. While it is noted that the majority of studies on technology 
adoption and usage in retail settings adopt a broadly positivistic perspective 
(Ha & Stoel, 2009; Huang & Liao, 2015; Pantano et al., 2017)—-often applying the 
Technology Acceptance Model (Davis, 1989), extending and combining it with 
other frameworks and constructs (Papagiannidis et al., 2017; Venkatesh, Thong, 
& Xu, 2012)—+tesearchers are increasingly stressing the need for more interpretive 
research designs in order to gain a richer understanding (Korpelainen, 2011; 
Rowlands, 2005; Williams, Dwivedi, Lal, & Schwarz, 2009). Indeed, technology 
adoption is a complex phenomenon (Rowlands, 2005) involving understanding of 
the unique points of view of the participants involved, human experiences and 
participant characteristics, the context in which adoption takes place, and the 
intricate and rapidly changing nature of technology (Pantano & Priporas, 2016). 
Consistent with this, the intention of this exploratory qualitative study is to gather 
rich and in-depth data to inform deeper understanding of the phenomenon inves- 
tigated, rather than generalising more broadly into other contexts. 


3.1 Augmented Fashion Store Development 


A sportswear fashion and apparel retailer located in central London (UK) was 
selected as a case study. This retailer sells sportswear and outerwear and an overall 
lifestyle concept. It occupies a premium market positioning, with an innovative 
brand image, due to the innovative materials used in products, the sports activities 
organised by the retailer, and in-store technologies. Moreover, the retailer focuses 
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Fig. 1 Example of a point of engagement, as visualised by consumers, welcoming customers, 
giving information and redirecting users to the retailer’s e-store 


heavily on providing an overall experience and creating a community among 
customers, in part by devoting a section of the premises to a café area, thereby 
providing a place where customers can meet, relax, work, and watch sports events. 

The store space was scanned and virtually reconstructed to realise a 3D model of 
the store, which consumers could access virtually. This allowed the development of 
immersive technologies, which were installed in the store for the purposes of this 
research. Three key products and a real model were scanned in detail, to visualise 
how a product would look when worn, as well as to increase the sense of realism of 
the experience. Tags or points of engagement were further inserted on specific items 
such as new products, at the store entrance to welcome customers, and on the 
in-store images of sports activities organised by the brand. From the highlighted 
items, tags would pop up by clicking on or walking by them, giving suggestions to 
the user and providing further information, linking and redirecting the user to the 
brand’s website and e-store, allowing customers to purchase items and view videos 
of sports events showcased. This served to immerse the user in the enhanced 
environment and space by interacting with it through the technologies used (Fig. 1). 
This 3D model can be used with all types of screens and VR headsets to help confer 
an immersive and enhanced customer experience whilst in the physical store. 


3.2. Procedure and Data Collection 


A convenience sample of 29 participants was recruited to take part in the research, 
which consisted of observation of participants trying out the technology and sub- 
sequent interviews (lasting approximately 30 min). The selection criteria were: 
(1) that participants were current loyal customers; and (2) were willing to take part 
in the research. The sample consisted of 27 male and two female participants, as the 
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brand’s target market is predominantly male-oriented consisting of 30—50-year-old 
men, successful, affluent, into a certain active lifestyle and loyal to the sport cat- 
egory. Three participants were aged from 20 to 24; seven from 25 to 29, two from 
30 to 34, one from 35 to 39, twelve from 40 to 49, and four from 50 to 60. In terms 
of frequency of shopping at the brand premises, seven participants visited the store 
once a week, eleven once a month, and twelve once every six months. 

Participants were first asked to try out the new technology on three formats— 
laptop, iPad screens and a VR headset—whilst in the physical store, and use each 
format to experience and autonomously explore the environment and immersive 
and augmented store experience. The technology included 3D, VR, headset, phones 
compatible with the headset, computers and iPads. Subsequently, participants were 
interviewed regarding their experience. The interview questions and themes for 
discussion were formulated to investigate perceptions of the degree of enhancement 
of the customer experience and the physical environment (i.e. shopping experience; 
level of immersion and engagement; degree of realistic experience; visibility and 
presentation of products and space), interaction with the technologies used (i.e. 
degree of realistic interactions; degree of immersion in the store space and envi- 
ronment; time in store; brand engagement; perception of the brand; purchase 
decisions), and further suggestions and considerations. Each participant was invited 
to talk openly and express their impressions and feelings in their own words. Notes 
were taken and typed into an online survey instrument to facilitate data collection. 
An established inductive process (Corbin & Strauss, 2008; Miles, Huberman, & 
Saldana, 2014) of applied thematic analysis (Guest, MacQueen, & Namey, 2012) 
was followed. The data analysis began with preparing and familiarising with the 
data, followed by an initial open and free analysis, exploring and identifying initial 
codes and sub-codes. As data analysis proceeded iteratively (Spiggle, 1994), codes 
were refined and then grouped into initial themes and categories. Further analysis 
then revealed three core themes: enhancement; interaction; behaviour. These 
themes form the structural and discursive basis of the following presentation and 
discussion of findings. 


4 Research Findings 


4.1 Enhancement: The New Store Environment 
and Shopping Experience 


In terms of participants’ perception of the store environment enriched with immersive 
AR technologies, most participants (n = 26) commented that this enhanced their 
shopping experience. Participants appreciated the new in-store experience, and con- 
sidered the enriched store to be entertaining, engaging, immersive and enjoyable. 
Some respondents said the immersive technologies contributed to make the in-store 
experience highly customised, which helped the retailer create a closer relationship 
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with the individual customer and further enhanced perceived service quality and 
experience (Pantano, 2016). Overall, observations and interviews showed that par- 
ticipants were generally optimistic about the 3D models and the augmented store, and 
the consequent enhanced in-store experience, and also revealed positive disposition to 
future developments of this form of store and innovative technologies. 

However, three participants did not regard this new type of store as contributing 
to their shopping experience. This can be linked to participants’ degree of accep- 
tance and usage of innovative technologies, which depends on several factors, 
including perceived usefulness (PU), perceived ease-of-use (PEOU), consumers’ 
characteristics (i.e. level of cognitive innovativeness, level of education, age, store 
channel preferences), attitudes about the technology (i.e. degree of familiarity and 
understanding of how to use a specific tool, confidence in using it in public) and 
context constraints (i.e. time availability, sources of information, store crowding, 
technology location in-store) (Davis, 1989; Huang & Liao, 2015; McCormick et al., 
2014). Some participants stressed that, as these technologies were still quite new, 
they needed to get used to them, as they were not yet familiar with them, especially 
in a retail context: 


I have not used it [immersive technology] for browsing products before. Up until now the 
experience of it has been novelty games on the iPhone. 


This suggests a need for retailers to introduce this new technology gradually to 
potential users, educating them and promoting the new tool by providing all rele- 
vant information (e.g. trained staff in-store, in-store posters and signs) (Lee, Meyer, 
& Smith, 2012). 


4.2 Interaction: Consumers’ Mobility and Interaction 
with the Enhanced Store 


Most participants (n = 25) said they found the experience realistic, facilitated by the 
ability to walk around the enhanced store. Several were enthusiastic about aspects 
related to the visibility and presentation of products and spaces, where colours 
played an important role. Moreover, participants stressed the important role of the 
immersive technologies in enhancing product visibility (in terms of features and 
fitting), which they found very realistic. The enhanced store also helped users find 
what they were looking for, thus making the interaction informative (Antéblian, 
Filser, & Roederer, 2014; Huang & Liao, 2015; McCormick et al., 2014). 

However, four participants felt the enhanced store needed a better and easier way 
to show and interact with products: 


I would prefer it [enhanced store], but it needs an easier way to look at products. 


Hedonic motivations for interacting with the immersive technologies prevailed. 
In fact, participants stressed the immersive, engaging, interactive and entertaining 
aspects of the enriched store. In particular, they pointed out the desire to see this 
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technology being used to tell the ‘story’ of the brand in more immersive ways, 
making the in-store experience more informative about to the characteristics and 
values of the brand, and its associated sporting/brand community activities, instead 
of merely focusing on selling products. These hedonic drivers—accompanied by 
utilitarian drivers (i.e. product information, location, availability, being redirected to 
the retailers’ e-store to place an order to speed up service etc.)—underpinned 
participants’ evaluations of the augmented store, leading to perceived enhancement 
of their experience. This is in line with previous studies showing both hedonic and 
utilitarian value of immersive AR/VR applications (Huang & Liao, 2015; Olsson 
et al., 2013), although here results revealed a prevalence of hedonic motivations for 
interacting with the immersive and enhanced store. 

Overall, participants liked the interactive elements of the enhanced store. Being 
able to move virtually from one part of the store to another and see the space 
through the immersive technology, and then be re-directed to the brand’s website 
and get extra information or place an order, emerged as important aspects making 
participants’ interactions with the 3D models favourably-perceived and realistic. 
This allowed users to interact with other retail channels whilst in the physical store, 
as the adoption of innovative consumer-facing technologies (representing touch 
points between retailer and consumer) has made the online and offline worlds more 
interrelated (Bonetti & Perry, 2017; McCormick et al., 2014), and perceived by the 
customer as integrated and as part of a unique, seamless experience (Verhoef, 
Kannan, & Inman, 2015). 


4.3 Behaviour: The Influence of the Enhanced Store 


Many participants found the enhanced store entertaining, and commented that the 
interaction with the innovative technologies and the enhanced store made them feel 
engaged with the brand (n = 21), and enhanced their perception of brand value: 


It’s engaging and puts you right in the centre of the experience. 


This would encourage them to spend more time at the enhanced store interacting 
with the immersive technologies and store, and nine customers stated that the 
immersive technologies and enhanced store increased their desire to shop at the 
retailer. 

These reactions are in line with previous environmental psychology studies in a 
retail context in relation to the influence of consumer-facing technology on con- 
sumer behaviour (Dacko, 2016; Pantano, 2016; Papagiannidis et al., 2017), although 
the present study arguably advances understanding by examining the use of inno- 
vative and immersive technologies in the real-world context. Generally, participants 
found the enhanced store to be different, entertaining, innovative, professional, 
modern and trendy, whereby the technologies used serve to keep the brand 
up-to-date. Such innovative in-store technical elements are in line with the retailer’s 
identity, image and positioning, reinforcing how the retailer positions itself and is 
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perceived by consumers and competitors (Pantano et al., 2017), as the retailer also 
uses innovative technologies and materials for their products—as one respondent 
commented: 


It pushes the boundaries, I am not surprised, it’s the same idea as what they [the brand] are 
all about; they are innovative in the textile, high quality. 


5 Discussion 


Building on these exploratory findings, the following discussion suggests ways to 
enable the possible development of a new store form, enhanced through immersive 
AR technologies. Resonating with the notion of ‘augmented places’ mentioned 
earlier, a schematic framework for this proposed ‘augmented store’ is outlined 
below. 


5.1 Store Augmentation: The Development 
of a New Store Form 


Immersive AR technologies resulted in an enhanced shopping experience within— 
and environment of—the traditional store of the retailer in question, and overall the 
respondents in this research were satisfied and impressed with this augmented store. 
The discussion now considers some more practical implications of existing theory 
which has been generated in more experimental settings in this real-world context. 

The augmented store can be defined as a physical retail store augmented/ 
enhanced by innovative and immersive AR technologies. In this store form, the 
customer’s current perception of the real store space, environment and shopping 
experience are mediated and enhanced by the use of 3D models and virtual sce- 
narios that consumers explore and naturally interact with while in the real store 
environment. This augmented store is characterised by the combination of AR 
technologies with the real store environment, thereby integrating real and virtual 
objects, store space and environment. The augmented store (Fig. 2) thus extends the 
traditional and real physical retail space boundaries surrounding the customer (such 
as the physical limits of the store, the items physically present in the store, the 
information at consumers’ disposal and the way it is provided etc.). Here, con- 
sumers’ natural interactions with the store environment through innovative and 
realistic interfaces, space mobility and visibility constitute another key feature, 
offering new possibilities to see and interact with virtual objects not physically 
available in the real store space, enriching content, and thereby leading to con- 
sumers’ deeper participation and helping to confer a richer and more immersive 
perception of the augmented store environment and shopping experience, enter- 
tainment and enjoyment in the real physical store. 
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Augmented store characteristics 


— ™s 


Immersive and enhanced Natural, immersive, engaging 
customer perception of store ——> customer interaction with store 

environment and shopping environment and space in real 
experience in the physical retail time through innovative and 
space through AR technologies Ib realistic AR technologies 


More immersive, enriched and 
augmented customer perception of in- 
store shopping experience and real store 

environment and space 


Fig. 2 Key characteristics of the augmented store 


6 Conclusions, Contributions and Future Research 


This study has explored the integration of AR techniques within a physical store 
and has suggested ways to develop a new store form, the augmented store, by 
making the store enriched, enhanced, more accessible, entertaining and efficient for 
consumers. Results showed that participants perceived the new augmented store to 
be more immersive, entertaining, engaging and enjoyable. They found the inter- 
action with the store realistic, leading to an enhanced brand perception and further 
increasing brand engagement. 


6.1 Theoretical Contributions and Managerial Implications 


This research contributes to the existing literature in multiple ways. First, it extends 
the existing literature on augmented and immersive places (Carmigniani et al., 
2011; Chang et al., 2015; Oleksy & Wnuk, 2016) to a specific real-world context of 
the retail store, by providing knowledge on the shift towards an augmented store 
and outlining its key characteristics. 

Second, it contributes to the existing literature on HCI (Kjeldskov & Graham, 
2003; Rogers, 2004) by focusing on consumers’ interaction with the technology 
(consumer-computer interaction or CCI), thus extending existing research on a 
generic user’s interaction with technology in a generic place. In particular, the study 
focuses on consumers’ interactions with AR technologies in a real-world context of 
an actual store. This extends previous research conducted in simulated and con- 
trolled laboratory environments with simulated consumers (Pantano & Laria, 2012; 
Pantano et al., 2017; Papagiannidis et al., 2017), by investigating perceptions of 
current actual consumers of a retailer, their interactions with immersive AR tech- 
nologies in a real, physical retail store, and their reactions to the new and enhanced 
store space and environment. 
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Furthermore, this research has implications for practitioners. It unveils positive 
consumer reactions to the augmented store form, providing practitioners with a new 
perspective on a specific new technology to be successfully integrated within tra- 
ditional points of sale. Retailers willing to further engage with customers by 
enhancing their in-store experience should therefore consider types of immersive 
technologies, where these can provide entertaining, informative and engaging 
experiences. 


6.2 Limitations and Future Research 


The study also has some limitations. The selected retailer’s target market focuses on 
a particular demographic. Thus, the results do not reveal in detail how other con- 
sumer types would react to and perceive this new store form. Further research could 
involve different retailers with varied target customers, to identify and analyse 
reactions from a more diverse range of consumers, incorporating participants from 
different demographic profiles, to gain a more comprehensive understanding which 
would then help obtain the right balance of innovative technologies and traditional 
services provided in store, to better satisfy a broader range of customers. Although 
the research takes a qualitative approach, eye-tracking technology or retailers’ 
EPOS data could be used to link consumer perception to purchase behaviour, for 
example by exploring whether consumers would spend more time in store due to an 
enhanced experience, or whether it influences purchase intention. Finally, future 
research could investigate managerial perspectives of consumers’ reactions to, and 
perceptions of, the store enhanced with AR technologies. 
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V-Commerce in Retail: ®) 
Nature and Potential Impact crak 


Anouk de Regt and Stuart J. Barnes 


Abstract V-commerce is an emerging phenomenon that is gaining traction in 
marketing and business literature and is becoming specifically more prominent in 
content related to retail practices. However, interpretations and explanations as to 
what exactly v-commerce refers to and comprises are inconsistent. This paper 
addresses the fluid conceptualisation of the v-commerce terminology and advocates 
the usage of v-commerce terminology exclusively for referring to, and as an 
abbreviation of, virtual commerce—for which a unified definition will be proposed. 
Taking a business-to-consumer approach, the current implementations of virtual 
commerce in the retail sector, as well as the potential and future research impli- 
cations will be discussed. 


Keywords Virtual commerce - V-commerce - Retail - Consumer - 
Alternate reality 


1 Introduction 


V-commerce terminology in marketing and business literature is becoming 
increasingly prevalent. Various authors agree that v-commerce will “reshape the 
retail landscape” (Ango, 2016; McKone et al., 2017). However, the concept’s 
shared origin between academia, industry and the media combined with ubiquitous 
implementation, introduced ambiguity to the v-commerce terminology; which 
currently embodies different, and even contradictory, connotations contingent upon 
the various stakeholders (i.e., often due to contextual differences). At present, three 
different usages of v-commerce terminology can be observed: (1) A stream of 
authors, predominantly practitioner literature, that relates v-commerce to digitally 
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native vertical brands and vertically integrated commerce (i.e., vertical commerce; 
Dunn, 2016). (2) A stream of literature that associates v-commerce with virtual 
commerce (e.g., Jin & Bolebruch, 2009). (3) A final stream of literature that refers 
to voice commerce (Beaumont, 2017; Zhang, Liu, & Li, 2009). The following 
paragraphs will expand on the different interpretations of the v-commerce 
abbreviation. 


1.1 Vertical Commerce 


The development of Internet-based commerce and the competitive and fast-paced 
nature of global markets have revived retailer’s interests regarding vertical inte- 
gration—which can be described as combining two or more stages of production 
and/or distribution, that are usually separate, under a single ownership (e.g., 
Buzzell, 1983). Dunn (2016) coined the term Digitally Native Vertical Brands 
(DNVBs), describing brands that are born on the internet and are maniacally 
focused on the customer experience (a.k.a., v-commerce brands). DNVBs primarily 
use e-commerce channels for interaction, transaction, and story-telling and rely 
heavily on their loyal customer base and user-generated content to spread the word 
(for examples of DNVBs in different industries see Pixlee, 2017). Adhering to the 
concept of vertical integration, DNVBs bypass traditional supply chains and dis- 
tribution channels and implement direct-to-consumer models. This enables these 
online (niche) retails to offer consumers high-quality products at reduced costs 
while at the same time enhancing their product gross and contribution margins. 
Although there are potential downsides to a vertically integrated business model 
(e.g., supply chain complexities and difficulties in scaling; e.g., Dunn, 2016; Wertz, 
2012), rising investment activity and acquisitions (i.e., Unilever bought Dollar 
Shave Club for US$1 billion; Unilever, 2016) could usher in a new era of cult 
brand monotheism, e-tailers and vertically integrated commerce. 


1.2 Virtual Commerce 


A lexical definition of virtual is “not physically existing as such but made by 
software to appear to do so” (Oxford dictionary, 2018). In line with this definition, 
Javornik (2016) defined virtuality as a “media’s capability of showing virtual ele- 
ments or virtual worlds, as experienced by the user through immersion or telep- 
resence in the environment created by computer graphics or visual elements”— 
adopting an experience-based perspective (cf., Steuer, 1992). To facilitate immer- 
sive and interactive experiences, alternate reality technologies (Table 1) are uti- 
lized; which can be classified along a virtuality continuum (Milgram & Kishino, 
1994). 
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Table 1 Alternate reality technologies 


Terminology Definition 
Augmented Alternate reality technology that provides an enhanced version of the 
reality (AR) real-world by overlaying our existing reality with an additional layer of 


digital information, which can be viewed through a (connected) 
technological device—such as smartphones or Augmented Reality Smart 
Glasses (ARSGs) 


Mixed reality Alternate reality technology that facilitates the merger of, and real-time 

(MR) interaction with and between, digitally rendered and real-world data and 
objects through connected technological devices (e.g., mixed reality 
headset) 

Virtual reality Alternate reality technology that is characterized by generating real-time, 

(VR) immersive and interactive multi-sensory experiences situated in, and 


artificially induced by, a responsive three-dimensional 
computer-generated virtual environment—usually paired with advanced 
input and output devices 


Virtual commerce encompasses conducting commerce through these medium 
types. Therefore, this paper defines virtual commerce as: electronically mediated 
commercial transactions that originate from an alternate reality technological 
platform and involve either digitally-generated or real-world products and services. 


1.3 Voice Commerce 


Voice activated commerce pertains to user interaction with commercial platforms 
and applications that utilize natural language speech recognition to enable 
self-service transactions over the telephone and other connected devices (e.g., 
Dennis & Harris, 2003, p. 205)—as such, voice recognition technology substitutes 
the online point-and-click decision-making process by introducing spoken com- 
mand methods. This shift towards conversation-based e-commerce is influenced by 
technological progress, including, but not limited to, improvements in the fields of: 
artificial intelligence, cloud computing and machine learning. Alongside there is the 
consumer’s rising acceptance and comfort levels towards conversational user 
interfaces (i.e., smart speakers, such as Amazon’s Echo and Google Home, that are 
often linked and or controlled by virtual voice assistants like Apple’s Siri and 
Microsoft’s Cortana). Although voice commerce brings in new challenges for retail 
(e.g., privacy concerns could fuel increasing restrictions regarding access to 
walled-garden personal data accumulated from voice interactions, inciting tension 
among digital ecosystems and potentially raising the cost of platform neutrality; 
Gartner, 2016), it is likely that consumer demand for voice commerce will continue 
to rise as industry leaders proceed to innovate. 

All of the above-mentioned developments are expected to deeply impact the 
retail industry. However, to advance marketing research practices and increase 
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understanding regarding v-commerce within the public discourse, it is imperative 
that we clearly distinguish these advancements. This research advocates to ascribe 
the v-commerce terminology exclusively to virtual commerce; and use different 
prefixes for the other concepts, respectively vi—for vertical (integrated) commerce 
and va—for voice (activated) commerce. The underlying rationale is that 
throughout the years, commerce has consistently advanced by moving alongside 
and making use of technological revolutions (i.e., the internet); and most practi- 
tioners and academics agree that the rise of virtual and augmented reality can be 
considered as the next technological revolution (e.g., Steinicke, 2016, pp. 33-43). 
In addition, former one-letter commerce prefixes all refer to the medium type and or 
technology. Examples include: E-commerce, which can be defined as “the use of 
electronic means to exchange information and to carry out activities and transac- 
tions” (i.e., electronic commerce; Wyckoff & Colecchia, 1999); M-commerce “any 
transaction with a monetary value—either direct or indirect—that is conducted over 
a wireless telecommunication network” (i.e., mobile commerce; Barnes, 2002) and 
T-commerce “electronically mediated commerce through interactive digital televi- 
sion” (i.e., television commerce; Arroyo-Cafiada & Gil-Lafuente, 2016). Adhering 
to this pattern, v-commerce should relate to virtual technology. It can be argued that 
virtual commerce, like m-commerce before (Coursaris & Hassanein, 2002), should 
be regarded as a subset of e-commerce—although some people have phrased it to 
be the “next step” (e.g., Alexandru, 2017). 


2 V-Commerce and the Impact on Business-to-Consumer 
Retailing 


In 2010, Jones stated that “a host of technological and social forces needs to 
converge in order for v-commerce to flourish” (p. 56) and it can be reasoned that we 
are currently at that stage. The maturation of VR and AR technologies heralds a 
fundamental shift in moving from the internet of information towards the internet of 
experiences (i.e., in which experiences replace information as the basic unit 
of currency; Kelly, 2016), and it can be argued that therein lies the biggest premise 
of virtual commerce; the potential to transform online shopping experiences and 
provide a (near) real-world equivalent (e.g., Papadopoulou, 2007). This is primarily 
because a virtual commerce interface, if properly designed, can support natural 
shopping behaviour by providing a more personalized, immersive and interactive 
experience (e.g., Chittaro & Ration, 2000). From a societal point of view, the 
consumer shift towards the v-commerce will be mainly driven by the younger age 
cohorts, since they are technology savvy, known to associate with brands to express 
their identity (e.g., Saxton, 2005) and already spend more money on ‘experiential 
purchases’ (cf. Van Boven & Gilovich, 2003). 
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2.1 Current Implementations of V-Commerce 


The retailing industry is starting to transform due to virtual technologies impacting 
all stages of the retail ecosystem. This paper takes a business-to-consumer approach 
and focusses on “front-end” implementations of virtual commerce. V-commerce 
will change the meaning of “what you see is what you get” since it provides 
consumers with the possibility to experience and explore different features of a 
product or service before the actual purchase. Notwithstanding the interesting 
development perspectives for brick-and-mortal retailers, research by Chesney, 
Chuah, Dobele, and Hoffmann (2017) indicates that this could be a major disrupter 
for online shopping, potentially bridging the trust deficit that might have prevented 
people from making purchases in web-based e-tailing environments in the past. 
Various retailers are already experimenting with different implementations and 
although currently virtual commerce is still in the early adoption phase, it can be 
expected that in the (near) future virtual commerce will be implemented on a more 
global scale. Currently, two main implementations of augmented-reality interactive 
technology (ARIT; e.g., Huang & Liao, 2015) can be distinguished; namely, 
enhancing the consumers shopping experience by utilizing it as a product try-on 
technology and employing location-based intelligence to offer interactive, 
contextually-relevant (personalized) experiences. With regard to virtual reality 
technology, the two main implementations that can be identified are employing 
virtual-reality as a new sales channel and implementing it as a channel for brand 
building. 


AR Implementations The most obvious fit for try-on technology are retailers 
where fit is crucial (e.g., retailers selling clothing). However, it does not stop there. 
CaratLane, one of India’s largest online jewellery retailers, implemented this 
technology to introduce the world’s first virtual 3D jewellery Try-on app—im- 
plementing facial recognition and three-dimensional imaging technology to turn the 
user’s laptop or smartphone screen into a mirror (Business Standard, 2015). MAC 
Cosmetics implements this technology on location in their New York retail store 
(Stolyar, 2017; i.e., stationary virtual mirrors, a.k.a, AR-mirrors). Another example 
is the IKEA Place app, which lets consumers experience how true-to-scale furniture 
would look, and fit, in their house (IKEA, 2017). Currently, most AR apps require 
consumers to use their hand-held (mobile) devices. However, it can be argued that 
with recent technological advancements in the field of wearable augmented reality 
devices, and particularly Augmented Reality Smart Glasses (i.e., devices that are 
worn like regular glasses and merge virtual information with physical information 
in a user’s view field; Ro, Brem, & Rauschnabel, 2018), business potential and 
opportunities for value creation will continue to grow. Regarding location-based 
augmented reality, multiple apps exist that link augmented reality with GPS sys- 
tems, geospatial data techniques and location sensors of mobile devices (e.g., 
Streetmuseum, which provides historical information about landmarks in 
London; BBC, 2010). Utilizing this technology, Blippar (2017) provides a business 
implementation by employing rich media units (i.e., digital banner ads) that deliver 
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content and features designed to drive active consumer engagement and interest in 
products. Since this technology can be implemented in almost any camera- 
accessible web and mobile environment, it facilitates scaling exposure across 
audience segments. 


VR Implementations Providing the first virtual reality department store, eBay and 
the Australian retailer Myer provide a good example of implementing VR as a sales 
channel (eBay, 2016). Utilizing a smartphone based virtual reality viewer, con- 
sumers could inspect items (i.e., move, rotate, zoom-in), access real-time product 
specifications (e.g., product range, pricing and stock information) and finalize 
purchases through the eBay app. To enhance consumer uptake, Sydney residents 
could sign up for a special exhibition tour and 15,000 VR viewers were made 
available to consumers free of charge. Implementing VR technology for brand 
building can take multiple forms, virtual brand storytelling being the most obvious. 
A showcase example is provided by The New York Times who partnered up with 
Google to create a line-up of 360° virtual reality films that could be made available 
to their (print) subscribers. The first project (i.e., The Displaced, which focusses on 
the lives of three children whose lives have been uprooted by war) won them the 
Entertainment Grand Prix at the Cannes Lions International Festival of Creativity 
(Adweek, 2016). Other examples of VR brand building include: (1) Product 
demo’s, demonstrating product attributes, features and functionalities (e.g., 
Chevrolet VR experience; Chevrolet, 2018). (2) Live streaming of events (e.g., 
Topshop London Fashion Week; Campaign, 2014), and (3) co-creation 
(i.e., NIKEiD VR Studio; RGA, 2016). Overall, it can be concluded that 
v-commerce offers exciting and new opportunities for brands, businesses and the 
retail industry—opportunities that we are only just beginning to explore. 


3 Discussion 


At this point it seems unlikely that v-commerce will render well established 
m-commerce and e-commerce practices completely obsolete—although, tech exec- 
utives (e.g., Zuckerberg) have previously mentioned that alternate realities have the 
potential to become the next major computing platforms in the future (e.g., Heath, 
2017). Nevertheless, v-commerce does offer a new platform for consumer-brand 
interaction that can supplement existing digital and physical channels. Therefore, 
retailers that are pursuing a true omni-channel strategy should start to consider how 
v-commerce can become an integral part of their overall approach (e.g., as marketing 
and or customer service channel). Concurrently, the Marketing Science Institute (MSI, 
2016) has encouraged the academic community to gear their research efforts towards 
the delivery of integrated, real-time and relevant experiences in context. Therefore, 
future research efforts should focus on addressing: (1) Contextual differences 
(e.g., online vs. in-store settings). (2) Differences between consumer segments 
(e.g., acceptance, trust and brand-self connection). (3) Cross-channel effects, 
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interoperability and comparative effectiveness between different v-commerce chan- 
nels and/ or traditional advertising platforms, preferably relating it to market segments 
and or product categories. (4) Generating universal v-commerce measurement met- 
rics. (5) Establishing laws and regulations (e.g., regarding data ownership/privacy). 
We encourage investigations into these potentially fruitful avenues of future research 
and practice. 
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Literature Review: Current Focus, mais 
Underlying Themes and Future 
Directions 


Liangchao Xue, Christopher J. Parker and Helen McCormick 


Abstract This literature review reveals the current research focus, underlying 
themes and prominent research gaps in the Virtual Reality (VR) literature. 89 journal 
articles from the 22 years are thematically analysed in order to non-obvious reveal 
interconnections and themes, including research focus over time and underlying 
themes by research discipline. Over half of all papers focus on the need to understand 
the VR shopping consumer, yet no consensus exists as to what the optimal experience 
is or how to design effective v-Commerce stores. The most prominent research gaps 
are related to the unique HCI aspects in v-Commerce that influence shopping 
behaviours. The impact of this review is establishing the current challenges and future 
directions for academia in order to make v-Commerce a viable reality. Specifically, 
future research should mainly focus on develop human factor theory in VR shop 
design (i.e. social dimension, eye-tracking etc.). 


Keywords Virtual reality - Retail - e-Commerce - Human factors - 
Augmented reality - v-Commerce 


1 Introduction 


This chapter reviews Electronic Commerce (e-Commerce) and consumer technology 
focused literature in order to detect key gaps and limitations in the literature must be 
contrasted to key developments and proposed applications. Through this, future 
research shall be directed towards making Virtual Reality (VR) and Augmented 
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Reality (AR) viable platforms for e-Commerce beyond the two-dimensional limi- 
tations of screens. 

Previously literature reviews on Virtual Reality have been conducted in fields 
such as Education (Freina & Ott, 2015), Tourism (Guttentag, 2010), Military (Rizzo 
et al., 2011), Healthcare (Nichols & Patel, 2002), Clinical trial (Cook & Triola, 
2009), Therapy (Moreira, de Amorim Lima, Ferraz, & Benedetti Rodrigues, 2013), 
Urban planning (Simpson, 2001), Entertainment/Gaming (Connolly, Boyle, 
MacArthur, Hainey, & Boyle, 2012) and Engineering (Cecil & Kanchanapiboon, 
2007). However, retail, e-Commerce and Virtual Commerce (v-Commerce) are 
absent from contemporary VR/AR literature reviews, presenting a void in academic 
knowledge which needs to be addressed. 

The aim of this literature review is to direct research projects into as yet under 
explored to increase the efficiency and effectiveness of the retail research com- 
munity. To achieve this aim, three research objectives were created: 


1. We need to reveal the current focus of VR/AR research to appraise the research 
areas which are well understood and thus less suitable for innovative potential 
exploration. 

2. We need to expose underlying themes in the VR/AR research outside of explicit 
research focus to understand which disciplines are under researched relating to 
beneficial technological applications. 

3. We need to uncover the most prominent research gaps related to VR/AR to 
direct research projects that shall address critical issues essential to technological 
breakthroughs. 


In addressing the research objectives, this literature review makes the following 
key contributions: 


e Over half of all papers focus on the need to understand the consumer experience 
integrating with (Sect. 3.1), and shopping through v-Commerce, yet no con- 
sensus as to what the optimal experience is or how to design effective 
v-Commerce stores exists (Sect. 3.3). 

e The majority of papers include underlying themes relating to UX design and 
purchase behaviour motivators, despite the apparent technological focus of the 
majority of the literature (Sect. 3.2). 

e The most prominent research areas that need addressing are shown to be related 
to the unique HCI aspects in v-Commerce that effectively influence shopping 
behaviours in order to make VR/AR platforms financially viable (Sect. 3.3). 


2 Methodology 


The principle sampling method was protocol driven, outlining the academic data- 
bases, search terms and filters the outset of the investigation. To reduce the influ- 
ence of database bias, three databases were selected for their excellence in the 
academic research community; Web of Science, Google Scholar and Scopus. 
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Table 1 Literature review search terms 


Technology aspects Consumer aspects 

3D store Consumer perception 

Future of shopping Consumer reaction 
Interactivity Consumer satisfaction 
Technological innovation Consumer/purchase behaviour 
Technology retail Shopping experience 
V-Commerce Shopping motivations 


Virtual environment 
Virtual reality 

Virtual reality retailing 
Virtual shopping 
Virtual social 
Vividness 


The search terms used in the resource collection process has been identified as 
technology and consumer aspects; presented in Bonetti, Warnaby, & Quinn (2018), 
Dacko (2017), and Papagiannidis, Pantano, See-To, Dennis, & Bourlakis (2017) 
(Table 1). These were identified through a close initial reading of the most 
prominent VR/AR literature (Bonetti et al., 2018; Dacko, 2017; Papagiannidis 
et al., 2017). 

To reflect the current state of research into VR/AR, the literature review focused 
primarily on papers from 2016 onwards. However, inclusion of earlier work (2001 
onwards) was allowed for historic comparisons to be drawn. Therefore, 70% of the 
papers were from 2013 onwards. 

To discover more advanced literature than our preconceived notions of protocol 
driven search strategy would allow for, forward snowball sampling was applied; in 
line with recognised Literature Review guidelines (Jalali & Wohlin, 2012; 
Kitchenham & Charters, 2007). In this application, key papers on AR and VR were 
identified from the reference lists of significant articles, revealing important and 
influential material for the literature review. 

Literature collection took place between October “17 and February *18, resulting 
in 89 pivotal journal articles being collected. Starting with Web of Science, each of 
the search terms (see Bonetti et al., 2018; Dacko, 2017; Papagiannidis et al., 2017, 
Table 1) were sought, filtering by the criteria specified above. Selecting 
high-quality journals which review the literature, beginning with the most recent 
publications and work back in time. Papers deemed relevant were downloaded in 
PDF form and stored within NVivo 11. During the analysis of the journals, key 
cited papers were identified, before being found, critiqued, and subsequently 
downloaded. Once the snowball sampling trail had stopped being fruitful, the next 
search term was investigated in the same way. Following this, the additional search 
engines were checked for the search terms to ensure all key papers were identified. 
To keep up to date, continue searching the literature during analysis process. Only 
academic sources (journal articles, conference papers, periodicals and academic 
books) were accepted in the resource collection phase. This was to ensure the focus 
of the paper remained academic and grounded in scientific processes. 
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Thematic analysis was applied to the uncovered VR literature with the aid of 
NVivo 11. While manual coding techniques have proven successful in traditional 
literature reviews, the computational power of thematic analysis software helped 
gain a deeper insight into the data than otherwise possible. Resources were the- 
matically analysed for the papers focus, future recommendations made, gaps and 
limitations highlighted in the literature, key findings, models, participant and the 
technology discussed. All data from this literature review (codebooks, classification 
sheets and figures) that can be made public is available through: https://doi.org/10. 
17028/rd.Iboro.6177005. 


3 Literature Review Results 


3.1 Current Research Focus on VR/AR 


To reveal the current focus of VR/AR research, Fig. | shows the key areas of 
research discussed in the reviewed papers. These topics presented in the hierarchy 
chart give a picture of the current knowledge of VR/AR technology, retailers, 
customers and different shopping environment in the VR shopping field. 

The key outcome of this analysis is how most research focuses on the Consumer 
Experience (orange) and the human factors of Virtual Reality (Green). This is 
despite the literature search focusing on technology literature; see Table 1. While 
inspecting research distribution over time (Fig. 2) a convergence on the need to 


Fig. 1 Hierarchy chart showing research focus 
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Fig. 2. Heat map of prominent research focus over time 


understand the human factors of the v-Commerce consumer occurs from 2016 
onwards. These issues have been repeatedly highlighted for over 20 years, yet are 
still unanswered. 

Critically, 15 papers state consumers are positive about AR/VR because they are 
new technologies that producing a significant novelty effect (MeiBner, Pfeiffer, 
Pfeiffer, & Oppewal, 2018; Yim, Chu, & Sauer, 2017) and providing specific 
benefit such as efficiency and better shopping value (Dacko, 2017; Papadopoulou, 
2007), which further lead to increase their shopping motivation (Altarteer, 
Charissis, Harrison, & Chan, 2013). Recent market research revealed that 58% of 
customers consider shopping with digital technology will be more interesting 
(Perks, 2016), and 63% saying they expect such capabilities to change the way they 
shop (Parro & Santoro, 2015). While VR requires further improvements in terms of 
technical solution and operational business, as consumers are still perceive the 
virtual environment as being in the stage of development (Papagiannidis et al., 
2007): 

The second most important outcome is that 37% of all papers argue that retailers 
need to understand those factors that affect consumer perceptions. Specifically, they 
need to better understand the factors that influence v-Commerce consumers. 
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These have been classified into comfort, content (environment, information, pro- 
duct display and product features), functionality (accessibility, interactivity, per- 
sonalised service, vividness), media richness, perceived value (convenience, cost 
and trust), social networking and user experience (physical presence, product 
involvement and product perception). Therefore, these factors should be considered 
when developing VR shopping environment. 

The third most important outcome relayed to value theory of VR. To understand 
consumer behaviours and examine the nature of the experience offered, it is nec- 
essary to investigate consumers and retailers and compare different shopping 
channel (offline; online and VR). Regarding to value to retailers, eight papers state 
that VR shows great potential for retailers. As it is considered to be highly inno- 
vative and market leaders in the adoption of new technologies in the marketplace. 
As such, benefits to early adopters may be significant. Nine papers report that VR 
can also provide brand engagement and offer retailers a plethora of profound ways 
to educate the consumer about a brand, share their brand’s story with customers, 
reinforce brand values and customer loyalty. 

Regarding to value to consumers, 11 papers show that consumers’ motivation 
can be considered as either hedonic value (for fun) or utilitarian value (for effi- 
ciency). Therefore, VR may provide value for consumer through these two aspects. 
Consumers can stay at home to compare or choose a certain product; reducing time 
and effort while shopping. Additionally, hedonic values refer to the emotional state 
generating from the experience and may include all elements (e.g. colour, music 
and other design elements) that contribute to a state of pleasure. Shoppers can 
receive value from these experiences, which their satisfaction and loyalty will be 
affected accordingly. There are benefits and costs associated with not only the 
products/services being purchased online, but also the processes of obtaining them. 


3.2. Underlying Themes in the VR/AR Literature 


To expose underlying themes in the VR/AR research outside of explicit research 
focus, Fig. 3 maps the literature disciplines against key themes common throughout 
the texts. Critically, these themes are discussed within the literature as separate to 
the explicit research focus presented within Sect. 3.2. 

While key phrases such as shopping, online, shoppers and retail can be expected 
from the search terms of this review (see Bonetti et al., 2018; Dacko, 2017; 
Papagiannidis et al., 2017, Table 1), many of the phrases coming from the word 
frequency search are both unexpected and intriguing. Terms such as ‘service’, 
‘museum’ and ‘tourism’ demonstrate the considered application of VR/AR outside 
the retail or physical items and into the commoditisation of experiences. One of the 
most interesting terms is ‘design’. As fashion companies pursue of higher profit and 
market share though leverage VR shopping, future research is suggested to feed 
into the design and expansion of v-Commerce platforms. 
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Fig. 3. Heat map of underlying themes versus research discipline 


What is crucial though is that the majority of underlying themes centre on UX 
design or factors that influence buying behaviour. This contradicts the more 
widespread focus of research presented in Sect. 3.1. Consequently, while the lit- 
erature may appear to address a wide range of topics, a convergence on the mon- 
etary application of VR/AR in an e-Commerce context is of vastly higher 
prominence than development for experiential or technical benefits. 

It is intriguing how website design research indicates that comfortable page 
layout, efficient search engines, updated information, clear navigational structures, 
simple checkout procedures and user-friendly interfaces were important to con- 
sumers’ use of online shopping. These design features could also be applied in VR 
shop design. Similarly, the layout of store in the 3D environment is regulated by a 
different concepts because it operates under new store design conditions and cus- 
tomer navigation functions (Vrechopoulos, Apostolou, & Koutsiouris, 2009). 
Despite this research requirement being made a decade ago, the unique HCI theory 
required to address it remains elusive within the VR/AR literature. Thus it will 
increase commercial benefits through appropriate design of commercial premises 
(e.g., shopping malls Huang et al., 2016). 
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Apart from design, another critical term is experience. Studies show that user 
experience refers to consumer participation and the environment relationship that 
results from the consumer interactions within a product, environment and brand. 
These have a holistic nature concerning cognitive, emotional, affective, social and 
physical responses to the shopping environment. On the other hand, Kolko (2011) 
suggests researchers focus on designing experiences rather than designing artefacts. 
It is expected that experience design become today’s commercial art, for example, 
as Pine and Gilmore (1998) suggested that create a theme that combines different 
product demonstrations together into one staged experience. 

On the other hand, through exploring shopping environment of online and 
in-store, it is discovered that v-Commerce may have to make up for some of 
e-Commerce’s inadequacies. This is because e-Commerce cannot include physical 
and tactile factors, which lack of product perception, and consumers may unable to 
compare products from the quality, size and style. 

Therefore, in order to design effective v-Commerce shopping environments, 
designers should stand on the consumer’s perception to understand the character- 
istics of consumers’ acceptance of technology, keep abreast of technological 
innovation and market trends, etc. This is to help set up marketing and retail 
strategies to enrich and improve the consumer shopping experience effectively. 


3.3. Research Gaps with VR/AR 


To uncover the most prominent research gaps related to VR/AR, Fig. 4 shows that 
the gaps in the current knowledge of virtual reality and retail sector. These gaps 
presented in the heat map need to be addressed through future research. 

The most resounding calls for research in recent years are Consumer access to 
VR (7), Level of Interactivity in retail environments (9), reconsideration of tradi- 
tional HCI theory considering new interaction opportunities in VR/AR (8) and how 
to design effective v-Commerce environments (8). While the prominence of these 
research themes has increased in recent years, they have, without exception, been 
called for by academics for over 13 years. 

At a more holistic level, the most prominent questions in recent years relates to 
designing for v-Commerce. While consumer experience has enjoyed equally 
important prominence between 2000-2015, the reduction in authors calling for 
research into these areas suggests that the key questions in these areas have already 
been answered through existing research. 

One important deficiency in the literature is the lack of influence variables 
concern towards v-Commerce shopping. There are 32 papers called on researchers 
to examine these moderating variables, including age of user, the interaction effect, 
prior VR experience, purchase time frame and brand communication. Hence, future 
research will strongly focus on verify whether a number of independent variables 
that have been shown to affect consumer perception will have significant impact on 
virtual retailing. 
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Fig. 4 Heat map identified showing gaps in the literature against time 


Furthermore, 50% of all papers state that there are still major gaps in the 
knowledge related to VR shop design. First, only few studies incorporated social 
stimuli of the online environments as main constructs in the S-O-R framework. 
Therefore, the papers proposed further research should develop S-O-R in human- 
computer interaction environment and social dimension. Second, eye-tracking, as 
an important technology for VR shopping, the challenges and efforts put into 
mobile eye-tracking studies are seldom made explicit in academic literature. 
Therefore, we suggest researchers incorporate eye-tracking methodologies into 
further VR studies. Third, there is no literature about user interface and design for 
disadvantages shoppers, which would be a very interesting area to engage with. 

Additionally, there are many gaps lie in consumer behaviour and VR/AR-related 
studies and how they link to sales. It is suggested that more in-depth investigation 
of human interaction to expand existing knowledge about consumer response to 
interactive technologies, as well as understand consumer behaviours and motiva- 
tions within different sales channel. Moreover, Bonetti et al. (2018), Mei ner et al. 
(2018) and Moorhouse, tom Dieck, and Jung (2018) asked researchers to explore 
these technologies’ acceptance. 

To generate a bigger impact, 8% of papers also show that the experimental results 
generated in laboratories might differ from the real-world environment. Therefore, 
further research should request permission from real VR retailers to manipulate some 
of their stores’ variables in order to run a field instead of a laboratory experiment, it 
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would be very difficult but it is important for theory development (i.e. high external 
validity). To explore this further, future research should examine the impact of 
exposure to a Virtual Reality experience via pre- and post-usage measurements of 
the dependent variables. This also requires a longitudinal study in the future that will 
review this work and examine whether new outcomes are discovered and whether 
the immersive environment has become more ‘accepted’ and, thereby, participants 
perceive more experienced with VR. 


4 Discussion 


4.1 Current Research Focus 


The current research extensively investigated the potential of AR/VR as a tool for 
e-Commerce, whether consumers will have a positive attitude toward these inter- 
active technologies. Since VR/AR currently has more enthusiasts than experts, 
which suggest that although the technologies are available, most consumers have 
yet to fully in engage with it. Accordingly, studies explore the value theory of AR/ 
VR. Such technologies open up ample opportunities for retailers to connect with 
customers from their comfort home. Meanwhile, consumers benefit from both 
utilitarian and hedonic value. For example, reduce perceived crowding, increase 
user satisfaction, and then bring a competitive advantage for retailers. Through 
virtual try-on, reducing risk in customers’ buying process, retailers then benefit 
from less return rate. However, these papers point out there are many factors will 
influence consumers’ perceptions in v-Commerce shopping, including comfort, 
content, functionality, media richness, perceived value, social networking and user 
experience. 

These findings enable us to appraise the research areas which are well under- 
stood and thus less suitable for innovative potential exploration in how VR is going 
to take over shopping malls and will help in making better decisions. However, in 
consideration of fictitious possibilities offered by virtual worlds, this positive per- 
ception then may not available when consumers are looking to buy products that are 
actually in use (Papagiannidis et al., 2017). Furthermore, the majority of consumers 
were also concerned about the devices’ pricing along with peripherals, adapting the 
new technology, which concludes to be worried about the user friendliness of the 
complete system (Sharma & Bach, 2016). In addition, Papadopoulou (2007) argued 
that the procedure of shopping in the VR store is more time consuming than the 
conventional web, because they involved typing instead of clicking. Consequently, 
they are still perceive the virtual environment as being in the stage of development. 
The current knowledge still lacks understanding the unique human factors or 
v-Commerce scenarios well enough to design effective and efficient retail platforms 
which justify the high access price (retailer platform development and consumer 
investment in technology). 
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The industry implications for these outcomes is that within the recent positive 
consumer response of virtual reality, v-Commerce is predicted to become next 
generation market platform. Therefore, it strongly encourages retailers to adopt 
such innovative technology either in-store or online to gain competitive advantages. 
Before adopting VR, company need to critical assess consumers’ perception of the 
technology and ensure that dimensions that their customers view as being important 
are designed to be attractive to them. 


4.2 Underlying Themes 


This review proves that in the current VR/AR literature, the most prominent 
underlying themes are design and experience. The findings expose website design 
and virtual store design are important to home shopping. While the traditional retail 
store layout theory produces predictions that are not consistent with the virtual 
scene, aS consumers are able to reach any virtual shop directly. Thus, more 
consumer-friendly user experience should be further developed to influence con- 
sumer buying behaviour online. 

These findings enable us to understand which disciplines are under researched 
relating to beneficial technological applications in how current literature on VR has 
emphasised the retail and technological aspects of VR, but it has neglected the end 
user’s needs and problem. Yet, VR is increasingly used in retail sector, even though 
research has not been able to catch up with the trend from a design perspective. 
Pantano (2015) suggests these technologies will help create new marketing expe- 
riences. In particular, v-Commerce may combine the benefits of the electronic 
trading environment with the information richness of face-to-face trading through 
features that compensate for the loss of physical presence associated with 
e-Commerce (Chesney, Chuah, Dobele, & Hoffmann, 2017), such as absent of 
merchandising presentation, especially for clothing category (Kaplan & Haenlein, 
2009). Thus, the questions that might bring to future are how to effective and 
efficient combine the value of online and offline shopping environments to 
v-Commerce, and what are the added-values we can create in v-Commerce to 
improve shopping experience. 

The industry implications for these outcomes is that v-Commerce shopping 
should support the notion of providing a solution “design for your needs’. It is 
important for retailers to mix the value from different sales channels and bring it to 
build user-friendly VR shopping. 


4.3 Research Gaps 


This review demonstrates that the most called for (yet unaddressed) research areas 
are the lack of influence variables concern towards v-Commerce shopping, the lack 
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of knowledge related to VR shop design (social dimension, eye-tracking, disad- 
vantages shoppers’ considerations), the lack of understanding of how AR/VR link 
to sales, lack of understanding of VR acceptance for shopping and the experimental 
results generated in laboratories might be different from the real-world 
environment. 

Figure 4 reflects that many studies calling for research VR shop design, while 
design aspect has been the focus a lot of papers for retailing and technology 
discipline. This means VR as a fresh technology, even though it has been intro- 
duced for a long time, researchers have only been paying attention to it since 2013. 
VR shopping design is an emerging research topic from 2017, thus a more sus- 
tained attention and a well-developed system are needed. Designers need to con- 
sider various influence factors that might affect shoppers’ perception when 
developing virtual environment for retailing. These factors are not unique, but joins 
the growing body of research calling for better emotional and cognitive under- 
standing in the consumer in order to develop more effective, and emotionally 
seductive e-Commerce platforms (e.g. Parker & Doyle, 2018; Parker & Wang, 
2016). 

The industry implications for these outcomes is that VR is emerging in design 
aspects to meet the consumer need, including user interface design, user experience 
design, visual merchandising etc. Successful completion of this research will enable 
a new means to understanding UI design for VR shopping and provide practical 
implications for multidisciplinary fields. 


5 Conclusion 


This chapter set out to direct research projects into as yet under explored to increase 
the efficiency and effectiveness of the retail research community. Through thematic 
analysis of 89 pivotal journal articles, the main contribution of this chapter is it 
signposts a clearer framework for locating future research inquiry and it highlights a 
research agenda that may provide the catalyst for this process. 

The implication of this review is that retailers should be realised that the today’s 
shoppers may have high expectations from these immersive environments and, 
thus, they will require to adjust and take this into account when developing relevant 
strategies (Papagiannidis et al., 2017). It should be highlight that it is important to 
focus on store design features and user experience design to enhance the social 
dimensions of being a customer in a VR shop as well as the ‘utilitarian’ and the 
‘hedonic’ dimensions. They should also integrate the benefits of online and offline 
shopping, which consumers most rely on (e.g. personal advice, after-sales service, 
etc.). 

This literature review is limited in the scope of the VR search terms used (see 
Bonetti et al., 2018; Dacko, 2017; Papagiannidis et al., 2017, Table 1) and the 
number of sources used in the search. While computational software NVivo has 
allowed for the analysis of most sources than before, a greater focus on papers 
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before 2015 could provide a deeper insight into the research avenues that remain 
uncapitalized upon. Future research should focus on develop the fundamental 
human factors design theory for VR shopping environments, focusing on the 
lucrative fashion market, predicted to be worth £64 Billion by 2020 in the UK alone 
(Sender, 2016). This addresses the issue that until now, successful commerciali- 
sation remains elusive due to the lack of human factors theory to direct the inter- 
action design process. Therefore, we need to understand: 


1. The format of VR shopping experiences that consumers and retailers best 
respond to because this will allow designers to create virtual retail environments 
which efficiently encourage buying behaviours. 

2. The ways VR shopping can offer retail experiences of higher value to target 
consumers than current retail environments because this will allow designers to 
focus on the retail areas that may profit most. 

3. The barriers and enablers of VR adoption for fashion retailers because all 
commercially viable designs must overcome technological issues to diffuse in 
the marketplace. 
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Part II 
AR & VR Experience Design 


What We Don’t Know. The Effect ®) 
of Realism in Virtual Reality ‘ates 
on Experience and Behaviour 


Marnix van Gisbergen, Michelle Kovacs, Fabio Campos, 
Malou van der Heeft and Valerie Vugts 


Abstract This study addresses the question how realistic Virtual Worlds should be 
designed in order to create engaging experiences and stimulate ‘natural’ behaviour. 
Creating high realistic worlds is time consuming and expensive and it is unclear 
whether it is always needed. With the aim of gaining insights about questions 
related to presence and realism, an experiment involving 72 participants was set up 
upon a Virtual Reality cycling apparatus in which different levels of realism were 
created. Users were observed and evaluated regarding their experience (engage- 
ment, presence, naturalness and negative effects), awareness of realism and beha- 
viour. The results indicate that, despite that differences in realism were observed, 
differences in realism do not have an effect on experience and behaviour. There 
seem other variables involved that can affect the whole experience in an enough 
intensity to obliterate the effects of a better sense of presence and realism. In 
addition, an increase in a perceived higher level of realism seems to be depended on 
a congruent increase of different elements within the virtual world. Merely 
increasing the level of realism of one element does not alter levels of perceived 
realism of users. 
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1 Introduction 


As Virtual Reality (VR) technology, e.g. processing power and screen resolutions, 
advances rapidly, high realistic and high-fidelity worlds become possible in VR as 
well. It seems somewhat intuitive that a more realistic environment, able to achieve 
a higher fidelity with the actual environment which the VR is emulating, would 
elicit a better sense of presence and experience from the users. However, creating 
high realistic worlds is time consuming and expensive and it is unclear whether it is 
always needed in order to engage users and let them behave in a natural manner. 
VR studies seem to produce mixed results and indicate a more subtle and complex 
relationship between level of realism and experience. Moreover, the level of realism 
in most studies seem, despite hard work in development, relatively low compared to 
what is possible with technologies of today and what is used within the context of 
current entertainment films and games. In addition, none of these studies analysed 
experience (presence, naturalness, engagement and negative effects) and VR 
behaviour when the VR environment represents a real (city) environment in which 
the realism and resemblance is altered on multiple aspects. With the aim of gaining 
insights about questions related to presence and realism, an experiment was set up 
using a VR cycling apparatus developed within Breda University of Applied 
Sciences in collaboration with Atlantis Games. The virtual world represented a bike 
experience within a street in a Dutch city (Breda), which differed in level of realism. 
The subjects were observed and evaluated regarding their awareness of level of 
realism, the VR experience and physical responses to events within the virtual 
world. Several studies have been conducted concerning virtual reality and cycling, 
however, these were mostly focused on the topics of rehabilitation and exercise 
purposes (Hagen, Chorianopoulos, Wang, Jacceri, & Weie, 2016; Song, Kim, & 
Kim, 2004). This study tries to contribute to determine optimal levels of realism to a 
given VR application in order to optimize the cost and time needed to develop it by. 
As such we might avoid creating levels of realism that won’t be perceived by the 
users or do not improve their VR experience significantly. 


2 Theory 


Presence and realism are important concepts regarding VR experience (Rettie, 
2004, Yu, Mortensen, Khanna, Spanlang, & Slater, 2012). They also seem to 
become more important, as the technology of displays, miniaturization and com- 
putational power evolves making it possible to (re)produce more realistic Virtual 
Worlds. A combination of four technological dimensions (sensory, interaction, 
control and location) are expected to increase a sense of realism or presence and 
enhance an experience within VR (van Gisbergen, 2016). However, realism is a 
construct with many different meanings. On the one hand it refers to resemblance, 
in which realism is operationalized as reproducing something that is known and 
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familiar for the observer. The object, environment or event in that case also exists in 
the ‘real’, non-mediated, world. On the other hand, realism can refer to something 
that is perceived as real without any knowledge or reference to an object or event 
that is known to the observer. Even to the extent that the object or environment does 
not exist in the ‘real’ world (van Gisbergen, 2016). In this study we focus on the 
latter, meaning users “subjective sense of being in the place depicted by the Virtual 
Environment” (Yu et al., 2012) without knowing the (resemblance with the) real 
situation but instead reproducing a feeling in which the VR experience is like the 
real world (Rettie, 2004). However, it remains unclear how much effort should be 
put in creating high levels of realism (Bailenson et al., 2005). 

On the one hand theories, such as gestalt, claim that high realism increases 
experience and natural behaviour. Users will behave more real when the world is 
perceived as more real (Pertaub, Slater, & Baker, 2002). On the other hand, there 
are theories, such as the uncanny valley (Slater and Steed, 2000), that propose that 
too much realism (resemblance) brings about a very strong drop in believability and 
comfort and as such may be forcefully rejected by humans as a defence mechanism 
(Bryant, 2001; Dautenhahn, 2002; Reichardt, 1978). It has been acknowledged by 
Brenton, Gillies, Ballin, and Chatting (2005) that there is hardly any research 
regarding manipulated realism as an experimental condition, and further research 
needs to be conducted in this area. Studies that have been conducted show mixed 
results. Research on the effect of realism in VR worlds indicated positive effects of 
increased realism on presence, arousal and liking (e.g. Barlett & Rodeheffer, 2009; 
Bailenson & Yee, 2006; Bailenson, Yee, Merget, & Schroeder 2006; Ivory & 
Kalyanaraman, 2007; Nowak, 2001; Nowak & Biocca, 2001; Zanola, Fabrikant, & 
Céltekin, 2009), as well as no effects or negative effects of higher realism in games 
or avatars on emotions, presence and behaviour (e.g., Anderson, Rothbaum, & 
Hodges, 2001; Bailenson, Blascovich, Beall, & Loomis 2001; Garau, Slater, 
Pertaub, & Razzaque, 2005; Ivory & Kalyanaraman, 2007; Slater & Steed, 2000). 
This leads to the first research question (RQ1): What is the effect of realism in 
virtual worlds on user experience and behaviour? As it is unclear whether realism 
has an effect on experience, it also remains unclear whether it will affect user 
behaviour in VR towards more ‘natural’ behaviour. Leading to the second research 
question (RQ2): will a more realistic environment cause a more natural response in 
a virtual world compared to less realistic environment? Although the effect of 
realism on experience and behaviour remains unclear, we do expect that users will 
notice differences in level of realism even when they have not used virtual reality 
before, leading to the following hypothesis (H1): A more realistic virtual envi- 
ronment will cause a higher perceived feeling of realism than a less realistic virtual 
environment. 
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3 Method 


A between-group (low versus high realism) design experiment was carried out 
which included an online questionnaire after the VR experience. Figure | shows an 
overview of the research design and data collection method. 

The research took place at Breda University of Applied Sciences (June 2017). 
The VR gear was connected to a regular bike mounted in an intelligent trainer, able 
to interact with the virtual environment (e.g. back-pedal brake). The virtual envi- 
ronment was presented to the users by means of immersive goggles (an Oculus Rift 
DK2 head-mounted display). The bicycle was placed in the middle of a silent room 
(Fig. 2). 

The virtual environment used in each virtual ride could be switched between a 
low realistic and a high realistic one (Fig. 3). A video camera recorded all sessions 
to observe the behaviour of the participants and a researcher was constantly present 
in the room. Participants were invited in the room and had to sign the informed 
consent form and read the information sheet. They were orally briefed about the 
procedures of the experiment and we asked to act as natural as possible on the 


Post-test only design 
Group A Low realism 


questionnaire + 


un a — 


Fig. 1 Overview of the research design and data collection method 


Fig. 2 Bike connected to the VR system 
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Fig. 3. Low and high Realism VR environment. Source CycleSpex 


bicycle as if they were cycling outside. Each participant was able to observe the 
virtual environment for 30 seconds to become acclimated to the assigned virtual 
environment. The VR experience lasted for approximately two minutes. The online 
questionnaire took approximately 10-15 minutes. In the end, the participants were 
thanked and little treats were present to show gratitude for their participation. 


3.1 Materials 


The virtual world consisted of a simulation of a straight-ahead bike path within a 
city in the Netherlands (Breda). At the end of each VR cycle experience, a dan- 
gerous intersection with a traffic light was created. The traffic light would turn 
orange when a participant would approach the traffic light, and it would subse- 
quently turn red. After approximately three to four seconds the traffic light would 
turn green again. A low and high realistic world was created. Both conditions 
contained the same environmental sounds (e.g. wind and bird sounds). Also, the 
following items were the same: characters, billboards, bicycle, traffic lights, and 
leaves on the ground. However, the following items were more detailed and in 
depth in the high realistic condition: buildings, street, lighting, sky, lighting, tex- 
tures. In addition, only the high realistic condition contained street lights, shadows 
and graffiti. 


3.2 Participants 


The participant sample of the study consisted of 72 BA and MA students that were 
recruited at Breda University of Applied Science. The mean age was 23 years 
(SD = 3.465) and did not differ between groups. Per condition, 36 people partici- 
pated in the experiment following the criterion of a minimum of 30 participants 
used in many VR experimental studies (e.g. van den Boom, Stupar-Rutenfrans, 
Bastiaens, & van Gisbergen, 2015; North & North, 2016). The number of female 
participants was the same for each group (50%). In the low realism condition, there 
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were 10 people who originated from abroad, while in the high realism condition 
there were 8 foreign people. None of the participants were familiar with the bike 
route in the city. 


3.3 Measures 


Presence 

We measured presence using the ITC-Sense of Presence Inventory (ITC-SOPD 
scale developed by Lessiter, Freeman, Keogh, and Davidoff (2001). A scale 
commonly used to measure VR experiences (e.g. by Bafios et al., 2004; Bruce & 
Regenbrecht, 2009; van den Boom et al., 2015). The questionnaire consists out of 
four factors: Spatial presence, Engagement, Naturalness and Negative effects. All 
items were measured using a 5-point scale (1 = strongly disagree, 5 = strongly 
agree). Presence was measured using 19 items (using statements such as “I felt I 
could interact with the displayed environment’, 13 items were used to measure 
Engagement (e.g. “I felt sad that my experience was over’), Naturalness was 
measured using 5 items (e.g. “The displayed environment seemed natural”) and for 
Negative effects 6 items were used (e.g., “J felt disorientated’). For all four factors 
separately, the items were averaged to form a scale which showed reliable results 
for the low and high realism conditions: Presence (« = 0.86 and « = 0.90), 
Engagement (« = 0.65 and « = 0.75), Naturalness (« = 0.66 and « = 0.77) and 
Negative effects (a = 0.77 and « = 0.63). 


Overall Perceived Realism 

Overall realism was measured using two subscales of the German VR Simulation 
Realism Scale (Poeschl & Doering, 2013; Witmer & Singer, 1998). This scale has 
been used in many other studies (e.g. Fromberger, Jordan, & Miiller, 2014; Reuter, 
2015). The scale consists out of two items Scene Realism (5 items, e.g. “Reflection 
in virtual space seemed to be natural’, « = 0.70 and « = 0.82) and Audience 
behaviour (4 items, e.g. “Behaviour of virtual humans in the virtual environment 
was authentic”, « = 0.84 and « = 0.85). All items were measured on a 7-point 
Likert scale (1 = strongly disagree, 7 = strongly agree). 


Item Perceived Realism 

Perceived realism was measured for 15 elements within the virtual world such as 
shadows, traffic lights and cars using 7-point scales (1 = strongly disagree, 
7 = strongly agree, e.g. “The cars seemed very realistic to me’). The items were 
analysed separately as well as combined as one factor showing reliable results for 
the low and high realism condition (« = 0.87 and « = 0.92). 


VR Attention 
Attention was measured for the dangerous traffic light condition using three recall 
questions. The first question was whether the traffic light was noticed, the second 
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was whether the colour of the traffic light was noticed, and the third question was 
whether the colour change of the traffic light was noticed. 


VR Excitement 

VR Excitement was measured by means of the question: did your excitement level 
change when you approached the traffic light? (answer options were I did not see a 
traffic light, yes and no). 


VR Behaviour 

Observed VR behaviour was first measured by asking the participants what they did 
during the dangerous traffic light situation that turned red. Participants indicated 
whether they did not stop (because they did no see the traffic light or colour change 
or did see it but not stopped) or did stop cycling (as they observed the traffic light 
turned orange or red). In addition, using video footage and live observations it was 
checked whether the participants stopped or not. Also, VR behaviour was cate- 
gorized into an active level behaviour (1 = passive, 2 = active) as participants 
themselves are often unaware of their physical behaviour as it is difficult to detach 
themselves from the interactions they performed (Kumar, 2014). The footage was 
watched numerous times before interpretations were made. Two researchers coded 
the behaviour as either being passive or active based on physical movement and 
how they communicated when approaching the dangerous traffic light situation. 
Participants in the passive stage did not physically react (e.g., stop for traffic light) 
and/or spoke out loud about the traffic light situation while participants in the active 
stage reacted to the traffic light with a physical reaction (such as expressions of 
surprise, laughter or alertness) or by mentioning the traffic light situation. 


4 Results 


Presence 

The results showed no differences between the low and high realism conditions on 
all four factors of presence: Spatial presence (Mjy = 3.11, SD = 0.49, 
Mhpieh = 3.12, SD = 0.56, t(70) = 0.34, p = 0.74), Engagement (Mioy = 3.56, 
SD = 0.38, Mhish = 3.61, SD=0.41, t(70) =0.57, p=0.57), Naturalness 
(Miow = 3.31, SD = 0.64, Mpign = 3.39, SD = 0.70, t(70) = 0.50, p = 0.62) and 
negative effects (Mjoy = 2.37, SD = 0.73, Mhigh = 2.16, SD = 0.52, t(70) = 1.40, 
p = 0.17). 


Overall Perceived Realism 

The low and high realism conditions seemed not to be perceived as different in their 
amount of reflecting reality. No differences were found between low and high 
realism conditions for Scene realism (Mjow = 4.70, SD = 1.02, Mhign = 5.14, 
SD = 1.02, t(70) = 1.83, p = 0.04, 1-sided) but not for VR Audience behaviour 
(Miow = 4.08, SD = 1.32, Mhign = 4.24, SD = 1.35, t(70)=0.51, p=0.31, 
1-sided). 
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Item Perceived Realism 

Participants experienced differences in realism between the items within the two 
conditions. Although both conditions were perceived as real, the high realism 
condition was perceived as more real when combining the evaluation of all ele- 
ments that were altered in realism between the conditions (Mjoy = 4.15, SD = 1.13, 
Mhieh = 4.76, SD = 1.10, t(2) = 2.33, p = 0.023). Closer inspection reveals that 
five out of eight items were perceived as more real in the high realism condition: the 
buildings, the street, the lighting, the textures and the bicycle path. Shadows, Sky 
and cars showed the same direction, however not significant. All elements that were 
identical in both conditions, such as traffic lights and characters, were as expected 
not perceived as different on their level of realism from each other. 


VR Attention 

The results indicate no effects of realism on attention for VR objects within a virtual 
world. In the low as in the high realism condition, no differences were found in the 
percentage of participants that noticed the traffic light (Mioy = 81%, Mhigh = 78%, 
7, N = 72) = 0.08, p = 0.77), the colour of the traffic light (Mioy = 72%, 
Mhigh = 70%, 1°(2, N = 72) = 0.46, p = 0.79) and observed the traffic light colour 
change (Miow = 64%, Mhigh = 56%, 7°(2, N = 72) = 0.76, p = 0.68). In both 
conditions most of the participants were aware of the traffic light situation although 
many also missed the fact that the colour changed right in front of them. 


VR Excitement 

Realism changed feelings of excitement. More participants within the high realism 
condition felt a change in excitement when having to all of a sudden stop in front of 
a traffic light compared to the low realism condition (Mjow = 57%, Mhign = 78%, 
7°(3, N = 72) = 7.90, p = 0.04). 


Observed VR behaviour 

No differences were found between the low and high realism condition concerning 
behaviour in front of the dangerous traffic light situation. In both conditions most 
participants stopped in front of the traffic light (Mjow = 67%, Mhigh = 61%, 76, 
N = 72) = 2.68, p = 0.75). Almost none (3%) kept on cycling when seeing the 
traffic light was orange or red. This result was confirmed in the observations as an 
equal number of participants stopped in front of traffic light in both conditions. In 
addition the video footage showed no significant differences in percentage of active 
participants between the low (53%) and high (67% realism conditions ¥7(1, 
N = 72) = 1.44, p = 0.23). 


5 Conclusions 


The aim of this research was to gain a deeper understanding about the effects of 
different levels of realism in virtual worlds on VR experience and behaviour. Two 
levels of realism were created within a VR city cycling experience and contrasted to 
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each other. Participants were able to cycle, while sitting on a stand-alone bicycle 
with a VR headset on. As expected (H1), VR users notice differences in realism. 
More effort in creating high realistic worlds will be observed and seen by VR users. 
However, the results also reveal that merely increasing specific VR elements is not 
enough create a higher experience or change behaviour. As indicated in the study of 
Bailenson et al. (2006), to generate more engagement and excitement there also 
need to be congruency in levels of realism between elements that create a virtual 
avatar or in our case a virtual world. Indeed, participants in our study for instance 
observed that the characters used within the high realism virtual world were not as 
real as other elements. As such participants proclaimed that this stops further 
feelings of a high overall perceived realism of the virtual world even when putting 
more realism effort in those elements. If one ‘stays behind’ (in our case the char- 
acters) this hinders the overall perceived feeling of realism. This is for instance 
indicated in one of our results in which the high and low realism conditions did not 
differ on (character) human behaviour realism. 

On the other hand, a higher perceived level of realism also does not immediately 
result in more natural behaviour (RQ2). The low and high realism condition seemed 
to stimulate the same natural VR behaviour. Both levels of realism elicited the same 
looking behaviour, resulting in noticing the same elements within the VR world. 
Also, user behaviour was the same for the high and low realism condition. Meaning 
that in our study users in low and high realism conditions seemed to cycle in the 
same realistic manner, for instance by stopping in front of a red or orange traffic 
light that reflected a relative dangerous situation. An explanation for the similar 
behaviour in VR worlds that differ in level of creative realism, seems to be con- 
nected to a generic level of VR experience that is elicited. No differences were 
found in experience between a low and high level of realism measured by means of 
presence, negative effects, naturalness and engagement (RQ1). A reason for the 
similarity in experience might be that several elements within the VR world already 
create a high level of experience. As such increasing realism of some elements 
within the virtual word might not do much more on experience. In particular, the 
relative new and innovative experience of cycling in a virtual world, might already 
have created a large feeling of experience and presence. Indeed, a study conducted 
by Slater, McCarthy, and Maringelli (1998) demonstrated that the greater partici- 
pants would move their bodies, the greater the sense of presence was felt. This also 
explains why these results seem to contradict the finding of another study con- 
ducted by Slater, Khanna, Mortensen, and Yu (2009), which indicated that more 
realism in VR causes a higher sense of presence. As in the study of Slater et al. 
(2009), users were not physically active in VR. Being active in an innovative way 
in VR, like cycling through a virtual city, might create a VR presence ceiling effect, 
especially for those who have little experience in VR. Previous studies did indicate 
that new VR users are highly immersed and feel very present in virtual worlds often 
leading to probably unnatural VR behaviour such as observing a VR world more 
intense compared to experienced users (Stupar-Rutenfrans, Ketelaars, & van 
Gisbergen, 2017; Syrett, Calvi, & van Gisbergen, 2017). 
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The outcome that more realism not always leads to increased experiences and 
more natural behaviour might be good news for some VR applications. A very 
expensive highly realistic virtual environment does not always seem necessary to 
achieve results. For our case this would mean that the cycling VR application could 
be used to create and test new cycling roads without having to invest a lot of money 
in a high realistic world. The same might go for other VR applications developed in 
other areas such as health, retail or culture. And even within the domain of robotics, 
where experts also believe that a certain level of realism is needed to create 
meaningfull experiences (Dautenhahn & Werry, 2004). However, more needs to be 
investigated and theorized on in order to extend these results to other domains. 

First there is the question how real our environments were. Where are we on the 
realism continuum with these tested VR worlds? Of course much more effort could 
have been put in creating even more realistic words up to a level they use in the 
gaming triple A industry. And although with current VR technology that would 
maybe not be very useful (e.g., the resolution in VR is not as good as used in non 
VR simulations), VR technology developments go fast. It would be wise in past and 
future studies to somehow be clearer about the level of realism and fidelity used in 
VR worlds used for research in order to be able to better understand results acquired 
using VR worlds. However a judging system in how and where to position VR 
worlds on the VR realism continuum does not exist yet. 

In line with the level of realism, it is also unclear how to create virtual worlds that 
can be compared on level of realism for research purposes. The level of realism in this 
study was manipulated by means of altering elements (creating more fidelity) and 
adding realism in the environments by adding objects or elements (such as shadows) 
or even the way it is possible to navigate and manipulate the VR world (e.g. by adding 
more stirring options). It is unclear how different elements create realism and it might 
be the case that adding other elements or altering different elements (such as the 
characters) might create more differences or higher levels of realism. However, it is 
difficult to research the different roles of different elements or to not add (or leave out) 
specific elements as the congruency between the different elements is important to 
create high perceived levels of realism. More research should be conducted on how 
different elements affect realism in order to also better be able to compare results from 
studies that (different level of realism in) virtual worlds. 

Finally, a relevant possibility to explain the contradictions among the different 
studies could be the existence of unknown variables influencing the phenomena. 
Some of these variables could potentially overshadow the influences that a better 
level of realism would provoke. Preliminary studies indicate that delay and expect 
experience can influence significantly the perception and opinions of the users 
regarding the evaluation of VR applications. 

Although more studies are needed to extend the reach of being able to apply the 
outcomes of this study, it is clear that investing effort and costs in creating a higher 
level of realism to obtain a better VR experience and more natural VR behaviour is 
not always needed. VR users observe realism differences in virtual worlds but not 
always experience them differently or act different based on those (unconscious) 
observations and experiences. In order to be able to pinpoint ideal levels of realism 
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attached to the purpose of the VR world and the experience of the VR users, more 
research is needed as well as a joint effort to classify VR worlds used in research on 
level of realism. 
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Adapting Jake Knapp’s Design Sprint ®@) 
Approach for AR/VR Applications a 
in Digital Heritage 


Helen Southall, Maeve Marmion and Andrew Davies 


Abstract Modern digital devices offer huge potential for the delivery of engaging 
heritage experiences to visitors, offering a better visitor experience, higher visitor 
numbers, and opportunities for increased tourism income. However, all software 
development entails risk, including the risk of developing a product which few will 
want, or be able, to use. Identifying user experience priorities and problems at an 
early stage is therefore extremely important. This chapter describes work in pro- 
gress on a shortened version of Jake Knapp’s Design Sprint approach, and its 
application to designing VR/AR solutions for a specific heritage case study. 


Keywords Augmented reality (AR) - Virtual reality (VR) - Digital heritage - 
Digital tourism - Digital humanities - User-centred design - User experience - 
Design thinking - Usability 


1 Introduction 


This chapter presents work in progress on a new approach to the design of AR/ 
VR applications for heritage sites, based on a shortened version of the Design 
Sprint approach developed by Jake Knapp at Google Ventures (Knapp, Zeratsky, & 
Kowitz, 2016). A heritage-rich area around the River Dee at Chester is used as a 
case study. 

Modern smartphones allow the delivery of engaging, customised experiences 
direct to the general public, on devices that they already carry with them 
(Tussyadiah, Wang, Jung, & Tom Dieck, 2018). Applications previously requiring 
specialised hardware at each point of use can now be delivered via smartphone. 
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AR experiences do not usually require extra equipment, and VR is also possible, 
e.g. via a Google Cardboard headset. (Fabola, Miller, & Fawcett, 2015; Casella & 
Coelho, 2013) Other affordable and versatile digital devices, such as tablets and 
media players, offer options previously beyond the scope of smaller heritage sites. 
However, as Power et al. (2017) point out, technology which stops users from 
achieving their goals, or delivers a negative experience, is destined to go unused. 
A fast, effective method of developing and testing outline designs before com- 
mitting to expensive full-scale development is therefore required. 

Knapp’s Design Sprint schedule is a pragmatic tool for rapid production and 
testing of software design prototypes, designed to cut down the resources needed to 
produce the final application, and avoid expensive user experience mistakes. 
However, scheduling an expert team for the full five days of Knapp’s sprint format 
is expensive and difficult. A compressed three-day version is therefore proposed, 
which still covers the key phases defined by Knapp et al. (2016): understand; 
diverge; converge; prototype, test. The objectives of this study were: to compress, 
refine and develop Jake Knapp’s Design Sprint Approach to foster and facilitate 
small scale development; to design a tourism heritage experience able to develop 
new narratives, and to attract and engage new audiences; and to design and test a 
new AR/VR solution that meets the needs of diverse stakeholders. 


2 Designing Digital Solutions for Enhanced Usability 


Software project failures of all types cause wider problems, ranging from incon- 
veniences such as flight delays, to risk to life (Charette, 2005; Fitzgerald & Russo, 
2005). In the cultural heritage (CH) field, a software project which does not deliver 
required output can cause issues ranging from lower engagement by visitors/users 
at best, to a complete failure to produce a usable product at worst, possibly dis- 
couraging funders from backing similar projects in the future. As Still and Crane 
(2016) put it, “No product is cheap if it does not work for its intended users.” 

Charette (2005) describes the root causes of many such failures as a combination 
of technical, project management and business decisions, which interact in complex 
ways to increase project risk. Projects always entail risk (Bennett & AXELOS, 
2017), and early detection of such problems is critical. If they are not identified until 
the final system testing phase, or, worse, until the system has been released to end 
users, the costs are usually many times higher than if the mistake had been iden- 
tified when it first occurred. 

Despite the wide use of software development in the CH field, CH projects 
are still vulnerable to these pitfalls. Positively, this means that CH projects can 
make use of existing methodologies to overcome them. One such approach is 
Human-Centred Design (Fig. 1), as encoded by ISO 9241:210 (2010), which lists 
four activities required to establish user needs and requirements when developing 
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Fig. 1 The ISO 9241-210 lifecycle of human-centred design for interactive systems 


software systems. These are: to understand and specify the context of use; to specify 
the user and organisational requirements; to produce design solutions; and to 
evaluate designs against requirements. 

Several design approaches make use of the essential recommendations of ISO 
9241:210. One of particular interest to us is the “design sprint’ approach developed 
by Jake Knapp while at Google Ventures (Knapp et al., 2016). This approach draws 
on the ideas of design thinking, a 5-stage process for identifying and resolving a 
problem in a human-centric way (Design Thinking, n.d.; Rikke & Siang, 2018). 
Knapp’s design sprints follow the general principals of human-centred design, and 
also validate design proposals by involving typical users at a very early stage, 
before committing to full-scale development of the proposed system. Design 
sprints, as the name suggests, aim to achieve validation quickly. They also fit in 
with a wider trend in the industry away from traditional technology-centred design 
(TCD) approaches, and towards cyclical approaches such as Lean and Agile. 
According to Knapp et al. (2016), combining lean approaches with design thinking 
is a fast and cost-effective way of solving software design problems. 

While Knapp’s original design sprint approach encourages empathy for cus- 
tomers and designing with human needs first, the 5-day duration remains a barrier 
to adoption. Five days is a short time compared to the effort required for many 
software projects, but getting the required people together for the five consecutive 
days is often difficult. In 2018, Knapp and colleagues began promoting a 
“semi-official’ version of the process they call Design Sprint 2, which runs over four 
days rather than five (Courtney, 2018). In 2017, staff at the University of Chester 
had also recognised this problem, and proposed a compressed three day version of 
the design sprint process. The current study tests the feasibility and effectiveness of 
this version, in the context of a heritage-rich area around the River Dee at Chester. 
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3 AR and VR for Heritage and Heritage Tourism 


As technology has advanced over the past 50 years, so too have its applications in 
specific disciplines such as archaeology. Archaeologists soon recognised the power 
of technology for enhancing their practice and embraced its capabilities. In the 
1990s, this progressed to the discussion and use of specific tools such as GIS, AI, 
internet applications, interactive computer graphics, and virtual 3D modelling, 
followed by the creation of interactive, multi-user, virtual, augmented and mixed 
reality environments (Ch’ng, 2009). Technology in experiential archaeology, for 
instance, aims to ‘bring a distant world [the past] into a psychological reality’ and 
relates to the emergent concept of ‘virtual time travel’ (ibid, p. 459). 

Technology has also been adopted across a wider range of heritage-related 
contexts, including the management, interpretation, communication and presenta- 
tion of historic sites, places, museums and visitor attractions. Museums and CH 
sites have led the way in the use of technological innovation in tourism, and AR/VR 
user interfaces have taken a variety of forms including the use of small screens 
(including mobile phones and wearable devices such as smart watches/glasses), 
fixed or projected wall displays, overlays of information onto real museum artefacts 
or objects and more (de Carolis, Gena, Kuflik, & Lanir, 2018). These technologies 
enable museums and CH sites to engage new and existing audiences, improve the 
accessibility and relevance of their collections, and ultimately enhance the overall 
visitor/tourist experience (ibid). 

Whilst the CH sector has embraced advanced technology, there has been a skills 
gap between technology developers and those with the context-specific expertise to 
employ them (Ch’ng, 2009; Guttentag, 2010). Guttentag (2010) recognises that a 
potential barrier to adoption in tourism more generally stems from the fact that 
advances in technology often take place outside of this field, and hence beyond the 
scope or awareness of tourism organisations or destinations that might ultimately 
benefit from them. Such technological advances, which are neither led by nor 
designed for tourism experts, may lack clear opportunities or straightforward 
application (ibid). e-Tourism explores the advances in, and potential for, technology 
in tourism management from a variety of stakeholder perspectives (Buhalis, 2003; 
Buhalis & Deimezi, 2004; Buhalis & Law, 2008), and Smart Tourism seeks the 
integrated, strategic use of technology in destinations to offer personalised, 
enhanced and meaningful experiences for visitors and residents (Wang, Li, & Li 
2013; Buhalis & Amaranggana, 2014, 2015). However, there remains a potential 
disconnect between those experts well versed in technology and its potential for 
problem solving and experience creation, and those tourism stakeholders (in this 
case in heritage tourism) who have context-rich, industry specific challenges that 
could fruitfully employ, test, and capitalise upon the capabilities of the technology 
that exists. 

Despite this disconnect, authors have long understood that technology presents a 
major opportunity for tourism, (Williams & Hobson, 1995; Dewailly, 1999; 
Buhalis, 2003). From a CH and heritage tourism perspective, this opportunity is 
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particularly significant, as technology can help the CH sector meet its intrinsic aims 
of protecting, interpreting, communicating and presenting their heritage to diverse 
audiences in engaging, innovative, competitive and sustainable ways. The UN 
Sustainable Development Goals are seen as relevant to CH due to its “historical, 
social and anthropological value” and its critical role in sustainable development 
(Xiao et al., 2018) and hence finding sustainable ways to improve and/or manage 
access is increasingly important. Furthermore, there is an educational value in the 
use of AR/VR in CH. Digital curation can enrich the learning experience of dif- 
ferent aged learners, from children exploring new CH environments and contexts to 
help them construct new meaning, to students developing enhanced critical thinking 
skills and emotional connection to their subjects. As a result, there is also a wider 
social value in CH and advanced technologies, as wider and more engaged audi- 
ences create better understandings of the past, and help develop or renew shared 
cultural values (Schaper et al., 2018). 

A further rationale for CH sites making use of advanced technologies relates to 
their long and often complex histories, and the fact that many have experienced 
considerable change over time. As such, fixed, traditional interpretation and com- 
munication methods are not always appropriate or sustainable for heritage sites as 
they either impact on the aesthetic of the site, fail to capture the attention of their 
varied audiences, or both. Furthermore, it may be hard for visitors to appreciate the 
significance of a CH site, or independently identify or understand historically rel- 
evant features, given that such features may no longer remain visible in the land- 
scape (Younes et al., 2017). Advanced technologies such as VR and AR can enable 
visitors to become immersed in such sites, using a reconstructed version of a 
historic site or place to recreate what has been lost, and/or augment what exists, in 
order to facilitate an interactive experience for audiences (ibid; Fineschi & 
Pozzebon, 2015). Increased presence, or the feeling of being present in a VR 
environment, has been shown to lead to a higher level of enjoyment and emotional 
involvement, and may also positively influence the intention to physically visit a 
place remotely experienced through VR (Marasco, Buonincontri, van Niekerk, 
Orlowski, & Okumus, 2018; Tussyadiah et al., 2018). This suggests that VR may 
afford a ‘try before you buy’ opportunity to enhance the effectiveness of destination 
marketing (Tussyadiah et al., 2018, p. 141), or alternatively even protect vulnerable 
sites from visitation when deemed necessary by allowing access in a different form 
(ibid; Manghisi et al., 2018; Marasco et al., 2018). 

The use of interactive technology has the distinct benefit of allowing personal 
interests to shape and tailor an individual’s experience (Neuhofer, Buhalis, & 
Ladkin, 2015). Interactive technology based on agents, for instance, can allow for 
different levels of experience, dependent on specific user profiles (Machidon, 
Duguleana, & Carrozzino, 2018). These might include specialist CH audiences 
versus more general visitors, different ages, skill levels, language requirements, the 
extent to which a visitor prefers a straight or more fun interpretation, and so on. 
With diverse audiences, accessibility is important. For example, while technology 
offers many benefits for older adults, they may also be excluded from some AR/VR 
interfaces due to difficulties processing or understanding data, or problems 
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interacting with the software due to lack of experience with the technology 
(Peleg-Adler, Lanir, & Korman, 2018). Accounting for differences in human 
cognitive processing, perception and ability to understand visual and other data is 
vital, and should be accounted for in the design process as much as possible (Raptis, 
Fidas, & Avouris, 2018). Likewise, the differing technical requirements and output 
quality of (e.g.) smartphone-based mobile VR, and VR headsets tethered to per- 
sonal computers, must also be allowed for (Marasco et al., 2018). The extent to 
which virtual time travel, with realism, interactivity, avatars and AI agents, 
multi-sensory engagement and a minimal feeling of mediation is achievable or 
desirable (Ch’ng, 2009) warrants further research in a heritage tourism context. 


4 Case Study Context: Chester, UK 


The Roman city of Chester (Fig. 1) is a well-established tourist destination in the 
North West of England. Its Roman walls form the most complete circuit of a walled 
city in the country, outlining the site of the original Roman fortress, and are a 
popular tourist attraction in their own right (Chester: Explore the Walls, 2012; 
Chester City Walls, n.d.). There are many tourist destinations within the walls, 
including a fine medieval cathedral, and the half-timbered galleries of the Rows, 
which attract visitors for their shops as much as for their architecture and historical 
significance (Chester Cathedral, n.d.; The Rows, n.d.). However, a challenge for the 
city is that many visitors do not explore beyond the city walls. 


5 The Digital Dee 


The area around the River Dee at Chester, about half a mile from Chester city 
centre, is recognised as having untapped potential in terms of its natural, cultural 
and historic significance, much of which is not evident in the current landscape 
(Wilding, 1997). The area is bounded by two historic bridges—the Old Dee Bridge 
and Queen’s Park suspension bridge. On the city side of the river, important fea- 
tures include Grosvenor Park, St. John’s Church (Chester’s original cathedral 
church), the Roman amphitheatre, the riverside promenade known as The Groves, a 
comer of the Roman walls, and the building which housed Chester’s Edwardian 
hydroelectric power station. Near to the Old Dee Bridge is a medieval weir and fish 
ladder. On the other side of the river, beside the remains of Roman stone quarry, a 
reconstructed water mill marks the sites of medieval fulling and corn mills, and a 
Victorian snuff mill. Further upstream is Churchill House, which was built in the 
1930s as the British Army’s Western Command Headquarters, and is currently 
home to the University of Chester’s Faculty of Business and Management. The 
Western Command Headquarters included a large network of underground rooms 
and tunnels; these have fallen into disrepair, having gone unused since the 1960s, 
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but are of great historical importance, having hosted meetings between Churchill, 
de Gaulle and Eisenhower in the run up to D-Day in World War II. They are an 
example of the type of site discussed in Sect. 3, where VR or AR may offer an 
appropriate way for visitors to gain a fully immersive experience of the site, since 
physical access is dangerous and impractical. Overall, the goal of the Digital Dee 
project is to use digital methods to encourage both local and non-local people to 
visit, enjoy and learn about these interesting and historic features of Chester’s 
landscape. 


6 Testing the Three-Day Design Sprint Approach 


The goal of this study was to test the effectiveness of the 3-day design sprint 
schedule (Fig. 2) introduced earlier in this chapter. The Digital Dee heritage 
tourism project was chosen as a case study because it was at an appropriate point in 
development, and offered a suitable range of materials and opportunities. Data 
collection on the effectiveness of the 3-day approach itself was largely qualitative, 
consisting of daily reflections from members of the design team, post-activity 
reflections, and analysis of the sprint output. This was in addition to the feedback 
provided by the five testers on the prototype design produced by the design team 
during the first two days of the sprint. 

Substantial preparation was required before the sprint, including date selection, 
design team selection, ethical approval, venue booking, catering arrangements, and 
tester selection. Issues of availability and commercial confidentiality had to be 
managed when selecting participants. Other preparations including physically vis- 
iting the Digital Dee area, and amassing a collection of relevant photographs, plans, 
and other information likely to be needed by the design team. 

During the three day sprint, a hi-fidelity interactive software prototype (Fig. 3) 
was produced and moderated, and testing was conducted with a variety of local 
stakeholders. User feedback was then analysed, and refined prototypes will form the 
basis of future business and academic projects in the fields of digital heritage, 


Day 1 


Understand 


Decide and critique 
Day 2 © Pick the strongest, most preferred 


Decide 


Test or present 


Da3 SQ & 


Test eaten 


Fig. 2 Adapted design sprint schedule 
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Fig. 3. An overview of the high-fidelity screens for the proposed ‘Digital Dee’ mobile application, 
created using Sketch 


marketing and tourism. This user-centred design approach and sprint process 
ensured that both iteration and empathy remained at the heart of the project, with 
the aim of designing the system to work for the expected users, rather than 
expecting users to adapt to the product. 

The full design team consisted of seven people, which is the maximum number 
recommended for a design sprint (Knapp et al., 2016). All were members of staff or 
researchers at the University of Chester. The key design sprint roles of facilitator 
and decider were fulfilled by staff with ISO 9241 and design sprint experience from 
the Department of Computer Science and the Informatics Centre. The other team 
members were an application designer, a Ph.D. student with VR and AR expertise, 
and researchers from the university’s departments of Computer Science and History 
& Archaeology, and Faculty of Business and Management, who acted as domain 
experts in the fields of tourism, digital humanities, Chester history, and heritage 
interpretation. 

The working venue for the design team had space for seven people to work 
comfortably, with a large whiteboard, video display screen, wall-space for posters 
and visual materials, and a stock of essential stationery, including sticky notes and 
voting dots. The room was booked for the sole use of the group for the full duration 
of the sprint. Hot and cold drinks were constantly available, and buffet lunches were 
brought to the room each day. This helped to maintain concentration levels and 
avoid distractions. Laptop computers were used for research, design and authoring, 
and iPhones and Google Cardboard headsets for usability testing. An IPEVO 
Ziggi-HD Plus Visualiser/Document Camera was used to record the usability 
testing. Sketch (2018) was used to create testable prototype screen designs for a 
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smartphone app designed to introduce visitors to the heritage of the Western 
Command Headquarters and its underground bunkers. ReactJS, A-Frame and 
WebVR were used for building interfaces and a prototype VR panorama of 
Churchill House (formerly the Western Command Headquarters) (React, n.d; 
A-Frame, n.d; WebVR Rocks, n.d.). 

The three day sprint was completed successfully. The design team worked well 
together, with a good balance of skills, knowledge and personalities. From a very 
wide initial brief, the sprint process enabled the design team to refine a specific 
example to prototype for the testers. This prototype was created on Day 2, with 
some work running on beyond the normal working day. The biggest challenge for 
this sprint was completing the necessary preparation in time for testing on the 
morning of Day 3. 

Following design sprint guidelines, five individuals were recruited to test our 
prototype. This number of testers is specified because, typically, 85% of usability 
problems are identified after only five tests (Nielsen & Landauer, 1993). For 
comparison, it should be noted that TCD approaches such as the waterfall approach 
would not involve users at all at this stage, and as Nielsen writes, “zero users give 
zero insights” (Nielsen, 2000). Our five testers were chosen to be representative of a 
range of typical users and stakeholders expected for the Digital Dee project. They 
were: a local heritage expert with experience of opening Chester heritage sites to the 
public; a long-term local resident with a high level of local knowledge and com- 
munity involvement, and expertise in the performing arts; a short-term local resi- 
dent, originally from Africa, staying in Chester as a student, with some knowledge 
of local heritage; and two Cheshire residents with very limited knowledge of 
Chester heritage (acting as ‘tourists unfamiliar with the area’). 

Feedback from the testers suggested that superimposing directions and infor- 
mation about attractions in AR could encourage visitors to explore different areas of 
Chester. Testers’ perceptions of the team’s proposed use of AR technology to 
enhance the Digital Dee prototype were positive, although there is clearly a need to 
make the host system easy to learn to improve acceptance of the technology. 
Similarly, there is a need to explore ways to make VR interactions more intuitive, 
familiar and usable (e.g., some testers initially had trouble understanding how to use 
the VR headset). However, testers did find the VR technology engaging, and were 
excited about the potential for it being utilised in the context of Chester’s heritage. 


7 Conclusion 


All design sprints are unique, as they all focus on different design problems, involve 
different people and organisations, have different priorities, and different technical 
and financial challenges. This was a pilot project, where all of the design team 
(though not the testers) were from the same organisation, with a very broad initial 
brief, but a relatively low level of financial risk. As with the original five day design 
sprint schedule, a different group of people using the same schedule in a different 
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context would have got different results (Jezovit, 2016). These reservations 
notwithstanding, the following findings and recommendations followed from the 
team’s experience with this pilot project. 

Members of the design sprint team must understand the “sprint mindset’. Sprints 
are structured, time pressured, follow a process, and are goal-orientated, with 
everyone having a clear role. Commitment and concentration are required, but pay 
dividends in terms of high productivity and cross-pollination of ideas. The com- 
position of the team must balance specialist expertise and technical skills, and 
dialogue between sub-teams at strategic points is critical for maintaining momen- 
tum and cohesion. 

Regarding the three day schedule, this was certainly effective in this case. 
Advance planning is important (e.g., organising access to resources which are not 
available online, or from the sprint location), and the key design sprint roles of 
facilitator and decider are even more critical than for a five day sprint. Proper UX 
testing is vital to ensure reliable results, and must be built into the plan for the 
design sprint from the start. Due to the tight timescales, it is essential to establish 
realistic deliverables quickly, to avoid becoming too ambitious, and being forced to 
scale back due to time pressure. Even so, the scheduled breaks are important, as 
they help people to maintain energy levels. Design sprint tools such as ‘crazy 8 s’ 
and ‘dot voting’ encourage creativity and give everyone a voice, and collaborative 
digital workspaces allow efficient sharing of information. 

Overall, as well as validating a proposed design approach for a specific local 
context, the three day design sprint approach was shown to have great potential for 
use in the CH and digital heritage fields, particularly where new or unfamiliar 
technologies are proposed as solutions. 
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Part Ill 
AR & VR in Tourism 


Designing Valuable Augmented Reality ®) 
Tourism Application Experiences me 


Eleanor E. Cranmer 


Abstract Augmented Reality (AR) has gained increased popularity in the tourism 
sector, for its ability to create enhanced tourist experiences. This, coupled with the 
proliferation of technologies has increased pressure for tourism organisations 
design and add value to tourist experiences. Whilst much research attention has 
focused on the potential of AR, exploring areas such as adoption, acceptance and 
usability, there is a lack of research outlining guidelines for the design of valuable 
AR application tourist experiences. This study uses a small UNESCO recognised 
museum in the UK, to identify key value adding features for the effective design 
of AR tourism applications. Adopting a multi-stakeholder approach, fifty interviews 
were held with five stakeholder groups, revealing four AR design categories; visitor 
value, organisational value, stakeholder value, economic value. Findings identified 
these categories should be considered for the effective design and implementation 
of enhanced tourist experiences, bridging a gap in current research. 


Keywords Augmented reality - Experience design + Value creation - 
Value enhancement 


1 Introduction 


Technology use has reached the point it has become fully integrated into our 
everyday lives (Wang, Xiang, & Fesenmaier 2016). The unique characteristics of 
mobile technologies, namely ubiquity, flexibility, personalization and dissemination 
make it a useful tool for both tourism providers and consumers (Ukpadi & 
Karjaluoto, 2016). Emerging technologies offer more ways for tourism organizations 
to engage, attract, communicate and enhance the tourist experience. As a result of the 
increased use of technology, the sector has witnessed changes to travel behaviours, 
decision-making, information searching (Wang, Kim, Love, & Kang, 2013), use of 
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idle time, experience-documenting, sharing (Tussyadiah & Zach, 2012; Wang et al., 
2016), experience enhancement, value-adding services and unique experience cre- 
ation (Leue, tom Dieck, & Jung, 2014). Nowadays, tourists are increasingly 
dependent on the internet and smartphones to obtain information about unknowns 
(Selvam, Yap, Ng, Tong, & Ho, 2016). Therefore, it is crucial that visitor attractions 
and tourist organizations address these changes and begin exploring new ways to 
attract tourists (Jung & tom Dieck, 2017). 

AR is the digital overlay of information onto a users’ immediate surroundings 
(Rauschnabel, Brem, & Ivens, 2015), and has received much interest within the 
tourism sector as a method to create enhanced tourist experiences (tom Dieck & 
Jung, 2017). An increasing number of scholars have explored its potential in areas 
such as user requirements (tom Dieck, Jung, & Han, 2016), acceptance and 
behavioural studies (Rauschnabel & Ro, 2016), AR tourist experiences (Han, tom 
Dieck, & Jung, 2017), AR value from a stakeholder perspective (tom Dieck & Jung, 
2017), and the benefits of AR from an organizational perspective (Cranmer, Jung, 
tom Dieck, & Miller, 2016). Yet, a limited number of studies outline guidelines or 
principles for the design of enhanced AR experiences (SA & Churchill, 2013; 
Kourouthanassis, Boletsis, & Lekakos, 2015). Due to the emerging nature of AR, it 
is suggested the design of AR experiences presents a number of challenges and thus 
should be approached holistically (Kourouthanassis et al., 2015). Further, the 
number of successful implication use cases of AR applications are still limited (Han 
et al., 2017). Hence, this study aims to identify key areas for the effective design of 
enhanced, value-adding AR applications to create enhanced tourist experiences, 
adopting a multi-stakeholder approach to provide a holistic understanding. 


2 Literature Review 


2.1 Augmented Reality and Cultural Heritage Tourism 


AR’s ability to overlay information on the real environment has propelled it to 
become a popular tool to enhance the tourist experience. The pressure for tourism 
organisations to adopt modern technologies, such as AR, has reached the point it is 
now considered a necessity (Han, Jung, & Gibson, 2014). Researchers believe that 
to stay competitive, organisations must find new ways to create added value by 
enriching tourist experiences (Scarles, Casey, & Treharne, 2016). tom Dieck and 
Jung (2017, p. 2) confirmed “nowadays many destinations and organisations have 
either implemented or begun to consider the opportunities offered by this new and 
innovative technology to enhance the visitor experience”. 

Hence, an increasing number of applications have been developed to explore 
how AR can add value to the tourist experience, creating ‘info-cultural-attainment’ 
combining leisure, entertainment, cultural, educational and social experiences 
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(Palumbo, Dominci, & Basile, 2013). Previous studies revealed AR can improve 
education and interpretation (Casella & Coelho, 2013), tailor information to tour- 
ists’ specific preferences (Kounavis, Kasimati, & Zamani, 2012), increase inter- 
activity (tom Deick & Jung, 2015), entertainment and engagement (Xu, Buhalis, & 
Weber, 2017). It has also been praised for its ability to market and promote cultural 
assets (Selvam et al., 2016), and has been reported to create deeper and more 
‘exclusive’ experiences at heritage sites (Minazzi, 2015). Jung and tom Dieck 
(2017) proposed when used in this way AR can increase tourist attraction com- 
petitiveness. Recent studies also revealed AR increased dwell time, empowering 
tourists to personalise and deepen their experiences (Scarles et al., 2016), as well as 
facilitating co-production and increasing tourists’ intention to spend (Jung & tom 
Dieck, 2017). However, whilst the majority of studies report positive experiences 
(Yovcheva, Buhalis, & Gatzidis, 2013), AR has been found to affect tourists’ 
behaviour and interactions, reducing interaction among groups and proximity to 
exhibits (Lanir, Kuflik, Dim, Wecker, & Stock, 2013). Crucially, the availability of 
AR applications does not guarantee enhanced experiences (Jung & Han, 2014). 
Therefore, there is a need to further explore how to design effective AR applications 
to create positive and enhanced tourist experiences. 

Although there are a number of successful examples of AR applications in 
cultural heritage, they still represent a small proportion of the market (Tscheu & 
Buhalis, 2016). Concern has been expressed towards low user adoption, because of 
a lack of AR acceptance studies in cultural heritage (Haugstvedt & Krogstie, 2012; 
Olsson, Lagerstam, Karkkainen, & Waananen-Vainio-Mattila, 2012). Moreover, 
Jung and tom Dieck (2017) and tom Dieck and Jung (2017) acknowledge difficulty 
for smaller cultural heritage organisations to develop effective AR experiences, 
since large investments present too much risk without prior proof of concept. 
Nonetheless, implementation is considered feasible in terms of costs and content 
creation (Kasinathan, Mustapha, & Subramaniam, 2016). However, further research 
is necessary to identify how to best design value adding AR application experi- 
ences, reducing the risks for small organisations and strengthening the proof of 
concept. 


2.2 Augmented Reality Experiences 


Regardless of the fact that AR applications present much potential, existing literature 
outlining guidelines or principles for the effective design of valuable application 
experiences is scarce (Sa & Churchill, 2013; Kourouthanassis et al., 2015). It has 
been suggested designing AR applications presents a number of intrinsic challenges 
and thus should be addressed from a holistic perspective (Kourouthanassis et al., 
2015). Tristan et al. (2017, p. 2103) supported that “AR has a specific and unique set 
of challenges designers need to overcome in order for this technology and its 
applications to be integrated into workflows, tasks, and experiences more seam- 
lessly”. Furthermore, Nordam (2016) claimed failure to address usability in the 
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design of AR applications results in increased user errors, which reduces trust and 
undermines users’ perceptions of the technology. Therefore, it is considered 
imperative to understand multiple stakeholders perspective towards the use of AR in 
cultural heritage organisations to inform the effective design of value adding AR 
application experiences. 

At present, AR research has predominantly focused on the augmentation of 
human cognition (Webb, Vincent, Patnaik, & Schwartz, 2016), and interaction with 
AR applications. Tristan et al. (2017) identified whist the full potential of such 
applications remains to be seen, the negative consequences of inappropriately 
designed applications are significant such as; loss of situational awareness, cognitive 
overload, interrupted work flow causing poor performance and increased human 
error. It is suggested AR applications should be designed to provide practical ben- 
efits, which cannot be created via other forms of media (Olsson et al., 2012). Further, 
AR experiences should provide quality information to create enhanced experiences 
(Jung, Chung, & Leue, 2015). Yet, there is a lack of research exploring the necessary 
design components to create valuable AR tourist experiences. In a study of the 
heuristics of AR applications it was identified AR applications should; fit the user 
environment and task, form communication functions; minimise distraction and 
overload, adapt physical and virtual worlds, fit with the users physical and perceptual 
abilities and provide access to off-screen objects (Tristan et al., 2017). Whilst these 
are useful considerations for the design of AR applications, they do not focus on 
value adding content creation or the type of enhanced experiences AR should create 
to add value for both visitors and the organisation. 

Previous studies have explored AR user acceptance in cultural heritage tourism, 
(e.g. tom Dieck & Jung, 2018; Leue et al., 2014), cultural differences in adoption of 
AR (Jung et al., 2018), AR applications heuristics (e.g. Tristan et al., 2017) and 
generic principles for the effective design of AR (e.g. Sa & Churchill, 2013; 
Kourouthanassis et al., 2015), but a limited number adopted a stakeholder approach 
to understand how to best design value-adding AR application experiences, despite 
the fact it is crucial to incorporate stakeholders’ view for enhanced AR experience 
design. Moreover, Tristan et al. (2017, p. 2014) suggested “the growth and 
expansion of new AR capabilities needs to be informed by good design. We need to 
evolve our design methods to incorporate this expansion”. Therefore, this study 
aims to understand, from a stakeholder perspective how to design value-adding AR 
applications to create enhanced tourist experiences. 


2.3 Stakeholder Approach 


In a tourism context, stakeholder involvement and collaboration are important to 
create enhanced tourist experiences (Kourtit, Macharis, & Nijkamp, 2014; McCabe, 
Sharples, & Foster, 2012). The stakeholder theory developed by Freeman (1983) 
proposed that organisations are characterised and influenced by their relationships 
with groups and individuals. In tourism, a stakeholder is considered “any group or 
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individual who can affect or is affected by the achievement of the organisation’s 
objectives” (Sautter & Leisen, 1999, p. 313). More specifically in a museum 
context, external stakeholders are individuals or groups with the power to impose 
tules, thus internal stakeholders have a strong interest in satisfying such rules to 
maintain integrity (Kotler, Kotler, & Kotler, 2008). 

Stakeholder involvement, particularly understanding visitors’ perceptions is 
crucial to create enhanced and creative experiences in the design and development 
stages of new technologies (McCabe et al., 2012). Kourouthanassis et al. (2015) 
identified that a common reason for the failure of mobile AR application experi- 
ences is because they are often technology driven, and ignore users’ needs or 
address them in the latter stages of the development process. Thus, to design value 
adding AR application experiences, it is important to engage all stakeholders from 
the start, since their perception is a critical ingredient for decision-making (Yang, 
Shen, & Ho, 2009). Adopting a stakeholder approach was praised by McCabe et al. 
(2012) for the effective design and implementation of new technologies, as a 
method of reducing knowledge barriers by involving a range of perceptions. Hence, 
such an approach plays a significant role in addressing the concerns of a wide range 
of stakeholders when implementing and designing experiences involving innova- 
tive technologies (Hall & Martin, 2005). 

Of the few existing studies which identify key design components for new AR 
tourism application experiences, Kourouthanassis et al. (2015) identified they are 
often technologically focussed, ignoring users’ needs. This is considered a hin- 
drance to the successful implementation of valuable AR tourist experiences. Olsson 
et al. (2013) found that mobile AR user-centred design is challenging by virtue of 
the fact users of emerging technologies, like AR can find it difficult to express their 
needs because of a limited knowledge or understanding of its potential. Therefore, 
this study employed a stakeholder approach, to encompass the views of all indi- 
viduals and groups that had power, influence or a stake in the design of an AR 
application experience to add value to Geevor. A stakeholder approach helps 
overcome the challenges presented by a lack of guidance outlining design criteria, 
often considered a key reason for the delayed implementation of AR in tourism. 


3 Methods 


This study focuses on Geevor Tin Mine museum, in Cornwall, UK a small cultural 
heritage tourist attraction. UNESCO recognised and multi-award winning, Geevor 
attracts over 40,000 visitors a year. The museum experience does not currently 
involve technology, and as a publicly funded organisation, management are eager 
to explore the potential of AR to enhance the tourist experience. Similar to other 
small organisations, Geevor are unable to make large investments without prior 
proof of concept (tom Dieck & Jung, 2017). To overcome issues associated with 
technology focused design and create a value adding AR application experiences, a 
multi-stakeholder approach was adopted. A stakeholder approach will yield a 
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holistic perception of key design areas to develop valuable AR applications that 
create enhanced experiences and reduce the risk involved in investing in 
AR technology. Stakeholder analysis was performed, identifying five groups; 9 
internal stakeholders (G), 6 Tourist Bodies (B), 3 Tertiary Groups (T), 2 Local 
Businesses (LB) and 30 Visitors (V). The sample was chosen on the assumption 
that “the sample must be appropriate and comprise respondents who best represent 
or have knowledge of the research topic” (Elo et al., 2014, p. 4). Non-probability 
purposive sampling was employed to interview all stakeholder groups except vis- 
itors, where due to practicality, convenience sampling was used. 

A total of 50 interviews were conducted between March 2015 and February 
2016. In light of the exploratory nature of this study, a semi-structured interview 
approach was employed providing freedom for stakeholders to add to, and extend 
questions (Saunders, Lewis, & Thornhill, 2012). Such an approach is considered 
crucial to increase the quality of data (Gillham, 2005). To ensure all stakeholders 
had a proficient understanding of AR to participate, prior to interview, all were 
shown a short video demonstration of AR in a museum context and provided with 
an AR information sheet. All interviews were recorded and transcribed and data 
were analysed using content analysis. Table 1 demonstrates internal, tertiary, tourist 
bodies and business stakeholder profiles. The majority (60%) of visitors identified 
themselves as ‘very much’ or ‘much’ in regard to their technical savviness, which is 


Table 1 Stakeholder respondent profile 


Code Organisation Position 

Gl Geevor Trustee 

G2 Geevor Chair of Trustees 

G3 Geevor Marketing Officer 

G4 Geevor Learning Officer 

GS Geevor Mine Development Officer 
G6 Geevor Mine Guide 

G7 Geevor Curator 

G8 Geevor IT Manager 

G9 Geevor Mine Manager 

Bl Cornwall Council Cultural Programme Officer 
B2 Visit Cornwall Chief Executive Officer 
B3 Cornwall Museum Partnership Chief Executive Officer 
B4 Cornwall Museum Partnership Development Officer 

B5 (Freelance) Museum Marketing Expert 
Bo Cornwall National Trust General Manager 

Tl University of Falmouth University Lecturer 

T2 University of Falmouth University Professor 

T3 St Ives Secondary School Secondary school teacher 
LB1 Count House café Assistant Manager 

LB2 Geevor Shop General Manager 
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suggestive they are regular users of technologies. This was synonymous by the fact, 


that 83% owed a smartphone, and the majority of those who did not said they 
owned a tablet (see Table 2). 


Table 2 Visitor respondent profile 


Code |Gender | Age Visiting Visiting from Smartphone 
group with owner 
Vil F 18-24 Family USA Yes 
v2 M 18-24 Family USA Yes 
v3 M 45-54 Family USA Yes 
v4 F 45-54 Family USA No 
V5 F 18-24 Family Nottingham, UK Yes 
V6 F 45-54 Family Nottingham, UK Yes 
V7 F 25-34 Partner Hull, UK No 
V8 M 35-44 Partner Hull, UK Yes 
v9 F 45-54 Family Birmingham, UK Yes 
Vv10 M 35-44 Family Solihull, UK Yes 
Vil F 55-64 Friends London, UK No 
v12 F 55-64 Friends France No 
V13 M 35-44 Family Weston-Super-Mare, Yes 
UK 
vil4 F 35-44 Family Weston-Super-Mare, Yes 
UK 
V15 M 45-54 Partner Netherlands Yes 
V16 F 45-54 Family Netherlands Yes 
V17 F 55-64 Friends Staffordshire, UK Yes 
V18 F 18-24 Family Bolton, UK Yes 
V19 F 35-44 Family Bolton, UK Yes 
V20 F 35-44 Partner Toronto, Canada Yes 
V21 M 45-54 Partner Toronto, Canada Yes 
V22 F 45-54 Family Hertfordshire, UK No 
V23 F 45-54 Family Essex, UK Yes 
Vv24 M 45-54 Family Hertfordshire, UK Yes 
V25 F 45-54 Family Reading, UK Yes 
V26 M 45-54 Family Reading, UK Yes 
V27 F 25-34 Family Cheltenham, UK Yes 
V28 M 45-54 Family Cambridge, UK Yes 
V29 M 55-64 Partner Milton Keynes, UK Yes 
V30 F 55-64 Partner Milton Keynes, UK Yes 
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4 Findings 
4.1 Visitor Value 


Many studies have explored the ability for AR to add value to the visitor experi- 
ence, enhancing their interactions with the real-world. This was confirmed by 
stakeholders who recognised AR experiences improve interpretation, bring the site 
to life, whilst catering to different knowledge levels and interests. AR was perceived 
as a method to help visitors “digest” complex information and exhibits (T1). Local 
Business (B1) supported that AR would be an effective way to “tie it together” in 
reference to sporadic exhibitions and buildings across Geevor. Importantly, AR was 
considered a method to improve visitors ability to comprehend and access parts of 
Geevor that are currently inaccessible, such as the underground mining works. 

Visitors in particular praised AR for its educational abilities, identifying that it 
should be a key focus in the design of experiences, incorporating elements of fun 
and entertainment. Through the effective design of AR experiences, Tourist Body 
(B4) commended AR as a tool to engage “non-traditional museum audiences”. This 
was further supported by a number of visitors, such as V2 who felt AR would allow 
you to “see it and really experience it”. Tertiary Leader (T2) believed it would 
improve tourists social and situated learning experiences. In the same way, AR was 
regarded particularly useful to engage children, providing visual representations 
allowing them to see exhibits come to life. 

Research widely acknowledges the navigational and orientational benefits of 
AR. Stakeholders confirmed this considering AR a way to improve the efficiency of 
tourists navigating the site, whilst improving their comprehension of the site as a 
whole. Further, AR was considered a tool to increase awareness of facilities (e.g. 
café and shop), potentially increasing traffic and thus improving onsite businesses 
revenue generation. 


4.2 Organisational Value 


Improving organisational sustainability was revealed as important by all stake- 
holders. AR was identified as an effective tool to increase visitors dwell time, and 
intention to spend. Furthermore, a number of stakeholders considered AR would 
broaden the visitor appeal. Visit Cornwall (B2) commented this would engage more 
“generalist not specialist” visitors, targeting all market segments. Increased visitor 
numbers and engagement would increase tickets sales and revenue, ensuring 
Geevor remained economically viable and sustainable year-round attraction. 

T2 recognised AR as a way to bypass the need for visitors to pre-plan their 
participation in guided tours, whilst ensuring all visitors shared the same level of 
experience. Geevor IT Manager (G8) proposed “AR is the prefect substitute for 
people” offering many advantages as a tool to improve both organisational and 
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visitor experience sustainability. Thus, it was agreed that introducing AR tours 
would maintain the authenticity of the existing experience, whilst enabling future 
generations to share the same level of experience. 

The marketing potential and merits of AR are much discussed within literature. 
Supporting this stakeholders strongly acknowledged ARs potential to increase 
Geevors marketing presence, raising the profile of the site, and on a larger scale 
Cornwall as a tourist destination. It was suggested AR could give Geevor com- 
petitive edge, whilst helping to attract more generalist audiences as well as appeal to 
younger target groups. Simply, by implementing AR it was suggested that more 
tourists would visit, inspired by a desire to try the new technology and experience 
something unique (B1, G2). In turn, a number of visitors suggested this would 
positively increase word-of-mouth marketing, sharing on social medial platforms 
and likelihood to recommend. Tertiary Group (T2) commented that society are used 
to instant sharing, suggesting AR would inspire higher visitor numbers “based on 
new visits rather than repeat visits”. 

Moreover, AR was recognised as a way to improve efficiency; managing the site, 
communication and performing daily operations. In busy periods there is a lack of 
staff availability, therefore visitors can find it hard to navigate, interpret and 
understand the site. To overcome this, stakeholders suggested designing AR to help 
staff explain complex processes, with the use of 3D animations and diagrams to 
enhance understanding and improve the efficiency of explanations (G4, B1). In the 
same way, introducing an AR self-guided tour was considered a solution to provide 
interpretation to more visitors, whilst engaging different ages and interests by tai- 
loring content to an individuals preferences, and enabling visitors to return to, or 
repeat information (G1, G2). Similarly, external stakeholders supported that AR 
would supplement existing tours, providing interpretation to all visitors, thus 
overcoming challenges of limited staff availability during busy periods. Visitors felt 
AR would be a valuable alternative, creating memorable experiences (V4, V12, 
V16, V18, V23, V25). 


4.3 Stakeholder Value 


The preservation of knowledge using AR emerged as a key design area when 
developing new AR experiences. Ex-miners currently working at Geevor are 
praised for their ability to create authentic visitor experiences, incorporating their 
first-hand knowledge (B2, T2). Tourists are motivated to visit by virtue of the fact 
ex-miners create such experiences, thus preserving this knowledge for future 
generations is crucial. Stakeholders recognise ex-miners will not be working at 
Geevor forever and highlighted the ability of AR to preserve the integrity and 
authenticity of the experience they create and stories they tell “preserving that 
knowledge” (G5). Numerous visitors supported this, such as V7 who commenting 
“first-hand knowledge, that is key”. 
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It was recognised as important for AR experiences to be designed to secure jobs 
for existing staff. Stakeholders considered AR would increase job security and 
employment opportunities, because the more successful and profitable Geevor, the 
more profits available to invest into staffing. Internal stakeholder (G2) added that 
designing a value adding AR experience would help staff to “secure their jobs, the 
more money we have on site the more secure their jobs are”. Similarly, it was 
considered the design and implementation of an AR experience would help create 
further employment opportunities. 

Heritage value and community pride are fundamentally important to the Geevor 
experience and on a broader scale Cornwall as a tourism destination. AR was 
perceived as a tool to help reignite and maintain a sense of pride in the history and 
heritage of the area. Cornwall Museums Partnership (B4) suggested through the 
effective implementation of an AR experience, visitors appreciation of Cornish 
heritage would be deepened, creating a behavioural change and increasing aware- 
ness of the need to protect and conserve heritage sites. Similarly, the process of 
curating and designing an AR experience was predicted as a method to improve the 
local sense of identity and significance of Geevor (B1). Moreover, implementing 
and designing value adding tourist experiences was seen as a way to increase staff 
engagement, raising the profile of the site, and improving staff morale, whilst 
making daily tasks easier (G1, G2, G4, G5, G9). 


4.4 Economic Value 


Implementing an AR experience was perceived as a method to attract further 
investment and funding, bettering the site and local area. AR was suggested as a 
way to demonstrate Geevor were shifting focus from being a mine to a tourist 
attraction (G3) and that efforts were being made to secure the longevity and future 
of the site. Cornwall National Trust (B6) summarised “the better the experience we 
can give people in the whole area, the more people will visit, and the more the 
economy will grow and the more money you will have to invest back into con- 
servation of those areas”. This was further supported by T2 and B2 who felt the 
introduction of AR would illustrate Geevor’s intentions to enhance and add value to 
the visitor experience, ensuring the longer-term viability of the attraction. 

In addition to attracting revenue, implementing AR was considered a tool to 
increase and secure additional sources of revenue. Stakeholders were of agreement 
that creating an enhanced visitor experience would engage wider audiences, 
increasing visitor numbers and thus ticket sales. It was considered this would have a 
positive multiplier attracting more people to local areas, who would spend more at 
local amenities, therefore helping to sustain businesses and improve local infras- 
tructure. Yet, there was debate among stakeholders regarding the most suitable way 
to integrate AR, whether it be offered free or visitors were changed an additional fee 
for an enhanced experience. No AR revenue model solution was agreed, and thus 
required further research. 
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5 Discussion and Conclusions 


5.1 Discussion 


This study aimed to identify key areas to focus the design of valuable AR tourism 
applications experiences at cultural heritage sites. The research used a case study of 
UNESCO, award winning Geevor Museum, in Cornwall, UK. Stakeholders agreed 
that AR would add value to the existing tourist experience and identified four main 
areas that should be addressed in the design of AR applications to create enhanced 
experiences. Few previous studies have explored design features, outlining ways to 
create enhanced AR application experiences. To progress current understanding, 
this study revealed when implementing and designing AR application experiences, 
the prevailing focus should centre on four key areas; visitor value, organisational 
value, stakeholder value and economic value. 

The need to add value, creating enhanced visitor experiences has been identified 
by a number of researchers (e.g. Wang et al., 2013; tom Dieck & Jung, 2017). The 
present study confirmed the importance of designing AR applications to enhance 
existing visitor experiences, for example improving interpretation, catering to dif- 
ferent knowledge levels and interests, and creating enjoyable, exciting educational 
experiences. This was supported by Palumbo et al. (2013) who claimed AR should 
develop “info-cultural-tainment” experiences. It is also suggested that designing 
AR applications to create value for visitors is important to increase competitiveness 
(Jung & tom Dieck, 2017). 

Kourouthanassis et al. (2015) stated that because of intrinsic challenges asso- 
ciated with designing AR applications, it should be approached from a holistic 
perspective. Hence, adopting a multi-stakeholder approach was central in gaining a 
holistic understanding of ARs potential. This was confirmed by stakeholders, who 
supported that AR applications should be designed to create value for the both 
visitors and organisation. Cranmer et al. (2016) proposed AR added value for 
visitors and stakeholders, besides this, few studies have explored the design and 
implementation of AR to create value for organisations. Stakeholders were in 
agreement AR would broaden the visitor appeal, engage all visitor segments, 
increasing ticket sales and thus revenue. Whilst some past studies recognise AR can 
be designed to increase visitor numbers and ticket sales (e.g. Jung & tom Dieck, 
2017; Scarles et al., 2016), none have explored the specific value or benefits such 
opportunities create for the organisation. 

Similarly, studies recognising the importance of designing AR to create stake- 
holder value, in addition to visitor benefits, are scarce. Stakeholders revealed AR 
should be designed to preserve knowledge and ensure the continuation of the 
award-winning visitor experience for future generations. This confirmed Olsson 
et al. (2012) findings that AR design should create practical benefits. Moreover, in a 
study of AR application heuristics, Tristan et al. (2017) suggested AR should fit the 
user environment, tasks and offer communication functions. Stakeholders also 
agreed that AR should be designed to preserve knowledge, ensuring the authenticity 


84 E. E. Cranmer 


and integrity of the current visitor experience. Further, stakeholders confirmed AR 
would help improve communication across site, whilst enhancing the efficiency of 
managing and organising, for example crowd management and guided tours. 

This study also revealed AR should be designed to create economic value. 
Whilst previous studies discuss ARs potential to generate economic return, few 
identify areas of design to create financial gain. For example, stakeholders con- 
sidered AR a method to increase tickets sales and intention to spend, highlighting 
economic value as a key design area. Jung and tom Dieck (2017) confirmed AR has 
the potential to increase intention to spend from the context of AR value, but not 
from an AR application design perspective. Few previous studies identified the 
economic multiplier effect resulting from the effective design of AR application 
experiences as recognised by stakeholders. Further research is necessary to identify 
the most appropriate revenue model for the implementation of AR, since stake- 
holders expressed mixed opinions. Moreover, whilst this study identified some key 
economic benefits of AR application experiences, further research is necessary to 
understand ARs full economic potential. 


5.2 Theoretical Contributions and Managerial Implications 


This study makes a number of theoretical contributions, at present existing research 
has failed to identify key design areas to inform the development and creation of 
enhanced AR application experiences. Few previous studies outline principles or 
guidelines for the design of effective AR application experiences (Sa & Churchill, 
2013; Kourouthanassis et al., 2015). The present study extends current under- 
standing, through the identification of four design areas to inform future develop- 
ments of enhanced AR tourism application experiences. 

In many cases, previous studies have not adopted a holistic approach to 
understand complex stakeholder perceptions (Kourouthanassis et al., 2015). 
Therefore, the adoption of a stakeholder approach, advances current understanding 
of the effective design of AR application experiences from a stakeholder perspec- 
tive, providing a holistic overview. In addition, the present study deepens existing 
understanding of AR experience design guidelines. Tristan et al. (2017, p. 2104) 
commented “the growth and expansion of new AR capabilities needs to be 
informed by good design. We need to evolve our design methods”. The present 
study contributes to the existing pool of knowledge towards good AR application 
design by identifying four key areas of focus. The use of a stakeholder approach to 
provide a holistic understanding further strengthens this contribution. 

Overall, whilst some findings are supported in previous research, the context 
from which these were examined have not previously focused on the effective 
design of AR application experiences in a cultural heritage tourism. Thus, the 
present study extends current understanding of the effective design of AR tourism 
applications to create enhanced experiences and should be used to inform future AR 
development projects. 
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5.3 Limitations and Future Research 


Organisational value emerged as a key area for focus when designing new AR 
applications, however, because of the case study approach employed, it is suggested 
to conduct further research at other similar organisations to substantiate the find- 
ings. Stakeholders did not agree an optimal revene model for the design and 
implementation of AR. Whilst adopting a stakeholder approach yielded an 
important holistic understanding, however, it limits the findings to the case under 
investigation. Thus, it is suggested further research examines the perception of 
stakeholders from other attractions and tourism organisations, not limited to the 
context of cultural heritage. 
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Abstract Recently, there has been an increased interest in the opportunities of 
virtual reality (VR) for the enhancement of the tourism experience. However, few 
studies have qualitatively explored the experience from elderly tourists perspective. 
Latest technology needs to be relevant and easy to use for elderly users in order to 
be accepted. Therefore, this study aims to examine the impact of VR on elderly 
tourists’ experience. In order to achieve the aim, the present study conducted 23 
interviews with elderly tourists at a cultural heritage tourism attraction in the UK. 
Interviews were analysed using thematic analysis and findings revealed impacts of 
VR onto elderly visitors’ experience as well as requirements with regards to VR 
applications within cultural heritage sites. There has been an overall positive atti- 
tude towards the use of VR among elderly visitors and strong intentions to return as 
a result of experiencing VR. 


Keywords Virtual reality - E-Tourism - Tourism experiences + Elderly users 


1 Introduction 


Recently, there has been an increased interest in the opportunities of virtual reality 
(VR) for the enhancement of the user experience (Tredinnick, 2018). Numerous use 
cases emerged on how VR can be utilized as part of the tourism experience (Huang, 
K. F. Backman, S. J. Backman, & Chang, 2016). Through the digital recreation of 
content, tourists are able to experience unknown destinations and inaccessible places 
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which adds to the overall value of the tourism experience (Yung & Khoo-Lattimore, 
2017). Recently, Jung and tom Dieck (2017) found that cultural heritage attractions 
can offer children, elderly or disabled visitors with an enhanced experience by pro- 
viding VR experiences of inaccessible attractions. Furthermore, Tussyadiah, Wang, & 
Jia, (2017) revealed that VR can be considered a good substitute for virtually visiting 
endangered areas, making it ideal for sustainable tourism. Museums worldwide have 
started to provide its visitors with interactive virtual experience. For instance, the 
Salvador Dali Museum allows visitors to “enter” paintings and virtually see scenery. 
Other examples include museums that offer users at home the virtual museum expe- 
rience to bring culture and heritage to everyone (Unimersiv, 2016). However, 
empirical research on the importance of VR as part of the tourism experience for 
elderly is still scarce and further research is needed to fully understand the potential in 
terms of opening up new tourism business and experience opportunities. According to 
Quan-Haase, Martin, and Schreurs (2016), latest technology needs to be relevant and 
easy to use for elderly users in order to be accepted and therefore, tourism attractions 
have to play close attention to elderly users’ acceptance to ensure full adoption. 
Therefore, the aim of the present study is to examine the impact of VR on elderly 
tourists’ experiences. In addition, tom Dieck and Jung (2016) revealed the importance 
of exploring context-specific user requirements prior to implementing applications 
within the tourism domain. However, user requirement studies within VR and tourism 
are scarce, in particular from the elderly visitors point-of-view. Therefore, we fur- 
thermore aim to identify elderly tourist requirements for using VR. 


2 Literature Review 


2.1 Technology Enhanced Tourism Experiences 
in Heritage Attractions 


Tourism is a very dynamic industry and thus, tourism attractions have to constantly 
respond to changes in order to remain competitive (Neuhofer, Buhalis, & Ladkin, 
2014). Technology is thereby playing and increasingly important role as today’s 
tourists are used to using mobile devices pre, during and after trips. Consequently, 
tourism heritage attractions have started to explore the opportunities of imple- 
menting latest technologies to enhance the tourism experience (Han, Jung, & 
Gibson, 2014). Recently, studies found that mobile augmented reality (tom Dieck & 
Jung, 2016) as well as wearable augmented reality (Leue, Jung, & tom Dieck, 2015) 
strengthen the learning experience at cultural heritage attractions. Through the use 
of latest technologies, tourists experience an enhanced form of interaction with the 
attraction, which was found to be more memorable after the experience. More 
recently, the effect of VR was explored in the cultural heritage context (Jung, tom 
Dieck, Lee, & Chung, 2016; Tussyadiah et al., 2017). The latest importance of VR 
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for the tourism context is largely linked to the birth of devices such as Samsung 
Gear VR or HTC Vive which allow for users’ to have an immersive experience. 
Tussyadiah et al. (2017) found VR enhances tourists desire to experience places 
while tourism attraction have to create highly aesthetic VR experiences without 
distractions from tourists’ surroundings to ensure continuity during the experience. 


2.2 Virtual Reality in Tourism and Attitudes of the Elderly 


Technology is developing rapidly with the tourism sector being among the 
industries that are mostly affected by changing means of distribution due to its 
information intensity as well as the intangibility of products involved. A further 
trend is population ageing as particularly within Western Europe the increase in 
older people is supposed to have huge impacts on tourism attractions (Kotler, 
Bowen, & Makens, 2010). As a result considering the trend of emerging tech- 
nologies in conjunction with population ageing becomes apparent. Charness and 
Holley (2004, p. 418) identified the main barriers of technology adoption from the 
senior market including “access, motivation, ability, design and training”. The 
major problem of technological use by older people is the rapid pace of develop- 
ment. In particularly, it was identified that older people often have cognitive barriers 
with regards to understanding connections between computer, programs and the 
internet which, again, decreases the intention to use latest technologies (Charness & 
Boot, 2009). Quan-Haase et al. (2016) explored the phenomena of elderly tech- 
nology users and found an often “forced-adoption” imposed by younger family 
members who expect the elderly to engage with technology. Instead, it was iden- 
tified that elder technology users’ will engage if technology fits their preferences, 
affordability and ease-of-use (Quan-Haase et al., 2016). Especially in the context of 
a tourism mining attraction, VR technology could be considered ideal in order to 
allow elderly visitors to access the mine experience without the necessity of 
physically climbing down a mine. This raises the issue of accessibility and how 
technologies, such as VR, can be utilised to provide new opportunities for elderly 
tourism experiences (Jung & tom Dieck, 2017). Yung and Khoo-Lattimore (2017, 
p. 2) defined VR as “the use of computer-generated 3D environment, that the user 
can navigate and interact with, resulting in real-time simulation of one or more of 
the user’s five senses”. Quan-Haase et al. (2016) confirmed the importance of 
ensuring technology’s ease of use to ensure adoption by elderly tourists. Previous 
studies started to explore the adoption of VR for tourism attractions (e.g. D. tom 
Dieck, M. C. tom Dieck, Moorhouse, & Jung, 2018) however, the adoption 
behaviour by elderly is relatively unexplored. Therefore, this study aims to 
investigate the impact of VR on the tourism experience and VR requirements from 
elderly tourists’ point of view. 
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3 Methodology 


Research with regards to tourists’ attitudes (elderly tourists in particular) towards 
the use of VR in tourism has been limited. This study examines elderly tourists’ 
perceptions of VR as it links to consumption of tourism experiences, and the 
objectives are to: 


e Explore the impact of VR on tourist experiences 
e Identify elderly tourist requirements for using VR 


With a focus of understanding elderly tourists’ attitudes towards and perceptions 
of VR as linked to tourism experiences, an exploratory, inductive inquiry was 
launched. The qualitative methodology adopted, assists in better understanding the 
phenomenon at hand, as it is grounded in peoples’ experiences. 

This study was conducted at Geevor Tin Mine Museum which is a UNESCO 
World heritage site and cultural mining heritage tourism attraction in Cornwall, 
United Kingdom. Geevor is an old tin and copper mine from the Eighteenth century 
and has been preserved as the largest mining site in the UK. The museum’s target 
market is varied and encompasses school groups, families and elderly visitors. 
However, parts of the museum are difficult accessible (e.g. underground mines) and 
older visitors often struggle to have the full museum experience. Therefore, as part 
of a project, VR was thought to be a good alternative to provide elderly tourists with 
an enhanced experience. 

The aim of the study is to examine the impact of VR on elderly tourists’ 
experiences and therefore participants were recruited on site. Sampling was pur- 
posive and judgemental to ensure participants already had the motivation to visit the 
site, which allowed the researchers to assess whether VR had an impact on the 
actual experience (as opposed to intention to visit). Any participants below 60 years 
of age were excluded from the study. 

Participants were initially asked to experience a five minute VR application 
using a Samsung Gear VR Headset. The virtual experience show-cased 3 museum 
locations, including miner’s preparatory area/changing rooms, elevator shaft and an 
underground area. These three stories aimed to show-case visitors a typical daily 
journey of a miner; arriving at the mine, changing into working clothes and then 
descending into the mine. Sound was also an important part of the experience and 
users were able to hear miners talking as part of experiencing the changing rooms; 
going down the mineshaft involved the noises of metal and the lift opening and 
closing its gates. After participants experienced the VR application, they were 
invited for the interviews. 

All the participating interviewees were asked for approval to be involved in the 
study, and certain rights including their anonymity were guaranteed. In total, a 
number of 23 semi-structured interviews were conducted in June 2016, and lasted 
on average 20 min. The qualitative answers and the notes taken during the inter- 
views underwent thematic analysis, a method for identifying, analysing and 
reporting patterns (themes) within data (Braun & Clarke, 2006), in order to identify 
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key concepts expressed by the interviewees. These were then cross-checked for 
accuracy and validity by an academic moderator who was present during the 
research. This involved comparing notes to assist in the interpretation of data and to 
ensure the inclusion of key topics (Gibson & Brown, 2009). Saturation was 
achieved fairly early in the analysis and last interviews were confirmatory. 


4 Findings 


A total of 23 semi-structured interviews were conducted. All participants were aged 
over 60, 11 participants were female and 12 male. The results of the study indicate 
the perceived impacts of VR on the experience of visiting a heritage attraction by 
elderly users and their requirements for using it. These two themes were most 
dominant across all interviews and a number of sub themes emerged throughout the 
analysis as shown in Table 1. 


4.1 Impacts of VR on Tourism Experiences 


Participants viewed the VR application as an excellent addition to the real museum 
experience. All but one participant, found that the VR session in fact enhanced the 
actual tourism experience, in several ways. 


4.1.1 Link Between Past and Present 


First, it gave them an opportunity to compare and contrast the past and the present 
(P3: ‘Visiting the mine today is fantastic...it sparks your imagination of how it was 
for the people down there...but the VR makes you...feel how it was there...it brings 
you a step closer to what it was’). Whilst the actual visit to the museum today 
provides a better understanding of the place, space and conditions, miners were 


Table 1 Themes and sub-themes of VR experience 


Themes Subthemes 
Impact of VR on tourism experience Link between past and present 
Accessibility 


Sharing experiences 
Novelty 


Elderly tourists’ requirements for using VR Presence and Immersion 


Control 


Storytelling and Human Elements 
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working under, it remains a museum void of processes and interactions people had 
with that environment. The virtual reality experience however, provides visual cues 
of the environment as it were in the past; more ‘alive’, fully operational with miners 
performing their everyday duties in that environment. 


4.1.2 Accessibility 


Secondly, it provided value to those with reduced mobility. Parts of the actual site 
can be prohibitive to people with access needs, as it entails long and narrow 
corridors, steps and very low ceilings. Therefore, when moving, walking and 
bending are difficult to perform, the site becomes inaccessible. The VR application 
therefore, was perceived by the participants as a valuable addition to the real 
tourism experience, because it provided a glimpse of the spaces that would 
otherwise be unreachable. According to one participant: “P&: it is great value when 
the legs are weak...I can’t go all the way, but now I know what it looks like’. 

Some of the participants also claimed that because of the confined nature of the 
actual physical environment, they felt uncomfortable to take the risk of visiting due 
to feelings of claustrophobia (P21: ‘I wouldn’t like to go down am mine like that but 
that VR is fabulous...there is no fear...’). Some of the participants also claimed that 
although they were not usually suffering from claustrophobia, they still would not 
visit the actual mine because they were afraid of what would happen if they 
couldn’t finish the route (P10: ‘I don’t know what can happen to me or if I'll feel ok 
for the whole duration of the tour...I need to have the option to sit down or get 
out...if I don’t, I won't go...I won't take the risk’). 


4.1.3 Sharing Experiences 


Many of the participants visiting the museum were part of larger family groups. 
Some of the participants mentioned that using the VR experience was beneficial for 
themselves as well as other members of the family: ‘P/9: I liked the VR experience, 
but what I liked even more was that it gave me an opportunity to see the [grand] 
kids excited, and us bonding over it...it wasn’t just babysitting, it was educational’. 
Another participant mentioned: ‘P/4: My daughter-in-law is now 6 months preg- 
nant, and she can’t bend to enter the mine. It is not nice to leave one behind, but 
now at least when we talk about the visit afterwards, she can also feel part of the 
group’. The VR experience was therefore viewed by some participants, not just as 
an integral part of the visit, or even adding value to the visit, it was seen as the only 
alternative to an otherwise unattainable experience. The added value of the expe- 
rience is also resonating in the fact that allows the experience to be shared with 
other members of the group. 
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4.1.4 Novelty 


Also, many of the participants viewed the use of the VR application as ‘PI1: an 
attraction in itself’. None of the participants were familiar with VR applications, 
although some had 3D experiences in other attractions. For most participants, the 
VR experience was considered a novelty, and a tourism experience in itself. They 
felt that it enhanced their visit to the museum greatly, as it not only provided ‘P6: 
an extra layer of conversation’ but it also made the museum ‘P9: more worthwhile 
to visit’. This attitude is particularly interesting, as the participants who were 
already at the museum (already had the intention to visit) explained that the VR 
experience (that they were unaware of before the visit) enhanced the perceived 
attractiveness of the museum. They felt that the VR experience made the attraction 
more enjoyable and appealing to visit and re-visit. 


4.2 Elderly Tourists’ Requirements for Using VR 


4.2.1 Presence and Immersion 


One of the main points expressed by the participants refers to the sense of ‘pres- 
ence’ whilst experiencing the virtual reality application. Most participants com- 
mented in very positive terms that they sensed they ‘were there’ and felt surrounded 
by the virtual world. Many of the participants even considered that they visited a 
place rather than they watched a series of images. Those that considered they visited 
a place, and had high levels of experienced realism, suggested that noise cancelling 
headsets would assist into minimising the awareness of the real world and therefore 
enable the immersion into the virtual world, as a way of optimising the experience. 
Those that felt they watched images, also attributed it to being aware of the real 
world. For instance, noises and sounds from the real world were distracting from 
the experience, and therefore could not immerse themselves in the virtual experi- 
ence. However, for some participants, this simultaneous sense of virtual and real 
worlds was comforting. It provided a sense of security, reducing levels of stress and 
anxiety associated with engaging with an unknown experience. As one female 
participant stated: ‘P5: I liked what I saw but I also liked to know I’m still here... 
you know...it was nice to hear the gentleman’s voice...and also I could feel Jake's 
hand in mine...I knew I was here and it made me feel safe’. 


4.2.2 Control 


Users also expressed mixed views with regards to the element of control of the VR 
experience. Some claimed that they would have preferred to have more control 
over the pace and speed of the application. They claimed it would have been 
‘P11: ...helpful to be able to pause and take more time me to look at something’ or 
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“P13: ...speed up the elevator shaft part that was boring’. They claimed that being 
able to start/stop at will and speed up/slow down the pace would make the expe- 
rience more enjoyable because they would be able to decide how long they could 
spend on different sections according to the level of interest. On the other hand, 
there were those who preferred to relinquish control, and just observe the virtual 
environment as it appeared in front of them. Reasons for this preference included 
concerns of performance and competency of use [‘P9: ...J’m not sure I could 
handle it’, *P2: I would probably do something wrong and miss out on what I could 
have seen, or even destroy it’|. Lack of control was welcomed as a means to reduce 
anxiety which allowed to ‘P15: ...focus on what you see, rather what to do with it’. 
Some mentioned that the experience was particularly nice because it felt like 
“P3: ...you were on a conveyor belt’, and therefore had a chance to immerse in the 
environment. 


4.2.3 Storytelling and Human Elements 


One of the key findings regarding elderly tourists’ requirements of VR application 
is its use for storytelling. All participants agreed that one of the main positive 
features the VR application is its potential to deliver a story in a more visual, 
immediate, and lively way. They explained that core motivation to visit the museum 
in the first place, is to listen to the stories of the people that lived and worked there. 
The VR application ‘P16: ...goes beyond the narrative’ making it easier to ‘PJ6: ... 
remember facts in a story form’. For instance, they explained that the experience 
becomes invaluable, when ‘P8: ... you have a miner walk with you and tell his story 
of his one hour of preparation and what happened on a typical working day’. 

Participants also highlighted the importance of the inclusion of ‘human ele- 
ments’ to compliment the storyline, particularly with regards to engaging the 
senses. They explained that the VR experience on its own is predominantly a 
visually stimulating experience. They suggested that the experience could be far 
more immersive if attention was paid to other senses. For instance, incorporating 
sounds, smells and adjusting room temperature, would make the experience more 
realistic and convey a better interpretation of the authentic experience. The 
explained that the story becomes more exciting when visual cues are congruent with 
the environment (i.e. tools, hats and jackets lying around in the preparation area) 
and also when ‘atypical’ or ‘special’ events are described (i.e. view of the blasts/ 
explosions in the mine). Also, sounds such as ‘workers on the background chat- 
ting’, ‘rattling of the cage elevator’, ‘water running’, and ‘tools hitting the rock’, 
create an ambience that better represents the sense of a busy place (perhaps with 
vibrations). Participants explained that although the VR experience will always be 
different to the real experience, it can tell the story of the people in a most edu- 
cational, fun and engaging way. 
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5 Discussion and Conclusion 


This research has contributed to the growing body of literature, through the 
examination of the impact of VR on elderly tourists’ experiences. The findings of 
the present study are twofold. With regards to the impacts of VR, the study found 
that the vast majority felt a positive impact of VR onto their museum experience. 
VR is offering elderly tourists’ to reminisce about miners’ past and provides them 
with perceived mobility to experience the mine which would otherwise be inac- 
cessible. In addition, VR allows families to share the entire mining experience as 
previously elderly visitors were not able to have the same experience as their 
children and grandchildren. This was considered an essential aspect of an enhanced 
social experience. Finally, elderly visitors found that VR makes the overall visit 
more attractive and worth visiting again. With regards to requirements, storytelling 
came up as an important element of the VR experience, presence and immersion as 
well as control which was considered highly important when visiting virtual 
attractions. 

Theoretically, these findings add in-depth knowledge into elderly tourists’ 
experience of VR. Previously, there has been limited qualitative research on VR 
experiences in the tourism context with a specific focus on elderly users. The 
present study provides a theoretical foundation for further analysis revealing the 
importance of presence, experienced realism, control and involvement. 

Practically, tourism managers can draw important conclusions from these find- 
ings regarding the enhancement of their tourism offering for elderly tourists. The 
study has shown that VR is considered a viable alternative for the provision of 
content in order to enhance storytelling and increase accessibility. Previously it was 
unknown whether elderly tourists accept or reject latest technologies such as VR. 
However, this study has shown an overall positive impact of the VR experience on 
elderly visitors’ experience. 

The present qualitative study was conducted using one VR application in a rural 
tourism attraction. Findings are therefore not generalisable to the wider tourism 
context. Further qualitative research is required to compare and contrast findings. 
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1 Introduction 


The tourism industry accounts for a significant portion of the GDP of many 
countries. Due to increased mobility and the constant increase of tourists globally, 
the industry is pressured to remain competitive in a very hard-fought global market. 
Per Han, Jung, and Gibson (2013), tourist destinations that want to become a 
competitive need to invest in technologies and find ways to derive value from them. 
Therefore, leveraging advancements in technology to provide efficient services 
should be one of the top priorities for the industry (Garcia-Crespo et al., 2009). 
Because of declining visitor numbers in some countries, museums and cultural sites 
should be the forerunners in adopting new technologies to enhance visitor expe- 
rience which is currently often perceived as boring and not engaging. Furthermore, 
because of product customization and personalization, customers are constantly 
seeking meaningful personal experiences rather than standardized ones (Neuburger 
& Egger, 2017). 

As Garcia-Crespo et al. (2009) argues, “Currently, the tourism industry is 
demanding an ever-increasing level of value-added services in technologically 
complete environments, which are integrated and highly dynamic”. As a result, the 
introduction of emerging technologies in the sector benefits both, cultural heritage 
sites as well as tourists (Tscheu & Buhalis, 2016). 

Augmented Reality (AR) allows superimposing virtual objects into real-life 
views that does not require the detachment of the user from reality (Azuma, 1997). 
There is a growing interest by the research community in AR. This is seen by the 
constant increase in the number of scientific papers related to the field in the last 
years (Werrlich, Nitsche, & Notni, 2017). Although AR technologies have been 
around for many years, they have only recently shown potential for value creation 
in the tourism industry (Han et al., 2013). The increased interest in this field is 
unavoidably related to technological advances in mobile and wireless communi- 
cations and sensor technologies, leading to the widespread use of smartphones and 
other devices, allowing for the introduction of AR into the mass market (Yovcheva, 
Buhalis, & Gatzidis, 2013). 

Tourist destinations and service providers that are able to re-think their value 
creation process through investment and introduction of new technologies are more 
likely to enhance customer experience (Tscheu & Buhalis, 2016). Moreover, the 
introduction of AR can increase the entertainment level of tourists concerning 
information access and the education process at specific sites of interest (Kecke8 & 
Tomicic, 2017). Cranmer, tom Dieck, and Jung (2018) argue that technology has 
re-shaped business models in the tourism industry by affecting the behavior, 
information searching and decision-making process of tourists. Gartner Inc. esti- 
mated that in the upcoming decade AR would have a significant influence on the 
market (as cited in Barsom, Graafland, & Schijven, 2016). The potential of AR to 
influence the tourism industry has been noted by many researchers. 

This paper starts with a literature review on the effectiveness of AR in the 
tourism industry, by identifying the benefits, challenges, and summarizing existing 
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studies in the field. Furthermore, through a case study at the Augusta Raurica 
archaeological site we compare the effects of AR on the experience of different user 
groups, namely digital immigrants and digital natives by applying the experience 
economy framework. Consequently, the results are analysed and discussed in detail. 

Our motivation is to assist the research community and tourist destinations in 
better understanding the prevailing differences between different user groups with 
different needs/priorities, in order to develop tailored and more engaging content for 
them. 


2 Literature Review 


2.1 Effectiveness of AR in Tourism—Benefits 
and Challenges 


There are many applications with AR capabilities have been developed around the 
world for the tourism industry. The strategies behind the analyzed apps are to serve 
as tourist guides, to recreate the past, or to give a glimpse into no access zones of 
significant cultural value (Boletsis, n.d.; Donovan, 2014; “EHC”, n.d.; Kim, 2010; 
“See what is not (yet) there—With the NAI and Augmented Reality”, n.d.). 

AR is particularly well suited for usage in the tourist industry because of its 
applicability inside four walls as well as in the open environment (Fritz, 
Susperregui, & Linaza, 2005). The ability to receive new information through AR 
while visiting a cultural heritage site or exhibit improves the learning process and 
leaves an enduring impact on the visitor (Moorhouse, tom Dieck, & Jung, 2017). 
A study undertaken by Stielak, Skola, and Liarokapis (2016) found out that users 
adapted easily to AR applications at cultural heritage sites because they perceived 
them as simple and intuitive. Moreover, by delivering information through AR 
about different exhibitions and curator processes, museums can enhance the visitor 
experience (Neuburger & Egger, 2017). 

In addition, using AR in museums can provide a solution to issues arising from 
spatial limitations and the physical exhibitions of valuable objects, as well as 
hardware investments (Neuburger & Egger, 2017). 

Yovcheva, Buhalis, and Gatzidis (2012) identify the relatively recent introduc- 
tion into the market, limited coverage achieved and the inability to effectively 
compete against traditional means of information as some of the prevailing chal- 
lenges for AR in the tourism industry. Kounavis, Kasimati, and Zamani (2012) 
identify missing compatibility across operating systems affecting content accumu- 
lation, dependency on an internet connection for usage, and the high prices of 
mobile internet bundles as some of the key challenges. Additionally, processing 
information through AR might result in cognitive overload and confusion, 
depending on the complexity of the information (Wu, Lee, Chang, & Liang, 2013). 
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A fundamental challenge for the further development of AR is social acceptance 
and corresponding issues arising from it such as privacy and fashion aspect for 
wearables (Azuma et al., 2001). 

A study was undertaken by Jung, tom Dieck, Lee, and Chung (2016) in the 
Geevor Tine Mine Museum with 163 participants found out that the visitor expe- 
rience was enhanced by the use of mixed reality applications. Similarly, a study 
carried out with school-children by Moorhouse et al. (2017) at the Manchester 
Jewish Museum in the UK aimed at assessing whether the technology enhances the 
experience of visitors, found out that AR adds value to the museum and enhances 
learning as well as the overall experience of the visitors. Furthermore, another study 
carried out by Chung, Han, and Joun (2015) aimed at assessing whether AR has the 
potential to influence a visitor’s decision to re-visit a cultural heritage site, found 
out that the technology has the capability to do so. Moreover, a study undertaken by 
Neuburger and Egger (2017) at the Dommuseum in Salzburg, arrived at the con- 
clusion that AR enhanced the experience of the participants and has the potential to 
enhance the experience of museum visitors in general. 


3 Problem Statement 


A study carried out with the purpose of examining AR user experience in the 
tourism sector by Stfelak et al. (2016) analyzed general differences with regards to 
the perceived ease of using AR between old and young participants. However, the 
authors were not able to identify any studies specifically aimed at investigating 
differences in the augmented reality experience between digital natives and digital 
immigrants. 

Prensky (2001) categorizes users into two groups, the digital natives, born after 
1980 and the digital immigrants, born before 1980. Additionally, Prensky (2001) 
suggests that there is an unfillable gap between these generations and _ that 
tech-savviness is a born feature and cannot be learned. On the other hand, Helsper 
and Enyon (2010) and Wang, Myers, and Sundaram (2013) argue that besides age, 
additional factors need to be taken into account when assessing whether someone is 
a digital native or a digital immigrant. 

In line with this, we debate further on this topic and investigate on a specific 
distinction between the user groups, relying on Helsper’s and Enyon’s (2010) 
critique of Prensky’s (2001) digital natives and digital immigrants categorization. 
The main question is to assess whether any differences with regard to the overall 
AR experience between the two user groups exist. Moreover, we investigate if the 
user’s technical skills and technological habits influence their perception of AR 
have an impact on the effectiveness of AR applications. 
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4 Methods 


We chose the experience economy framework proposed by Pine and Gillmore 
(1998) for carrying out their case study, as it has already been used in several 
research papers focused on assessing the effectiveness of AR in enhancing customer 
experience in the tourism industry (e.g. Jung et al., 2016; Neuburger & Egger, 
2017; Radder & Han, 2015). 

Pine and Gilmore (1998) propose 4 realms of experience: entertainment, edu- 
cational, aesthetical, and escapist in which the visitor’s participation can either be 
passive or active and the relative degree to which they can be absorbed or immersed 
in the experience. With regard to the education experience, nowadays, tourists want 
to be actively engaged with the destinations they visit and want to educate them- 
selves about them (Falk, Ballantyne, Packer, & Benckendorff, 2012). In line with 
this, Fotaris, Pellas, Kazanidis, and Smith (2017) argue that AR’s potential for 
making the education process more engaging and immersive has been recognized, 
and stakeholders are looking for ways to utilize these competencies of the tech- 
nology in various sectors. With regard to the entertainment domain of the frame- 
work, tourists are willing to be entertained while themselves being passive and 
merely spectators during the process (Pearce, 2008). 

Besides, escapist and aesthetic realms are quite similar, as both of them achieve 
to immerse the visitor during the experience. The major distinction between them is 
the relative degree of participation during the experience. For example, rafting in 
the Grand Canyon would represent an escapist experience as it involves active 
participation, whereas standing over the canyon and overviewing it would be an 
aesthetic experience, as it does not require active participation (Pine & Gillmore, 
1998). 


4.1 Survey and Case Study 


In order to determine which user group (i.e. digital natives or digital immigrants) 
the participants belong to, a questionnaire was prepared based on the features 
proposed by Helsper and Enyon (2010). All the questions are formulated as 
dichotomous statements (except question 6). 


I multi-task when using different technologies. 

I have access to a range of new technologies. 

I am confident in the use of technologies. 

T use the Internet as a first port of call for information. 

I use the Internet for learning as well as other activities. 

I have the following number of ICT devices (mobile phone, computer, smart 
devices, etc.) in my household. 


OS oe Sho 
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7. use the Internet first for school/work information. 
8. I have good or excellent user skills in mobile and internet technologies. 


Using the survey, we categorized participants into user groups, by looking at 
additional factors besides age. Seven questions (Use of multitasking, Access to new 
technology, confidence in use, Internet first for information, Internet for learning, 
Internet for school or work and good or excellent skills) had to be answered 
positively to qualify as a digital native. Participants, who responded to one or more 
questions negatively were classified as digital immigrants. 

Afterwards, a structured Likert scale survey with questions formulated as 
statements about the four realms of experience proposed by Pine and Gilmore 
(1998) was prepared. The survey entailed three questions about each of the four 
realms of experience adapted from a study carried out by Jung et al. (2016). 


Education 


1. I was able to learn new things by using the AR app. 

2. The experience made me curious and I wanted to explore the topic in more 
detail after using the AR app. 

3. I enjoyed the overall learning experience facilitated by the AR app. 


Aesthetic 


4. I found the visual appearance of the content presented by the AR app attractive. 
5. I liked the level of detail provided by the AR app. 
6. I found the AR app pleasant and user-friendly. 


Entertainment 


7. I could focus on the content provided by the AR app without getting distracted. 
8. I was very entertained by the content provided by the AR app. 
9. The overall experience of using the AR app was fun. 


Escapist 


10. I felt I went back in time and place while using the AR app. 
11. I felt I was someone else while using the AR app. 
12. I was completely detached from reality while using the AR app. 


The authors went together with the participants at the site and cautiously 
instructed them through the AR installment. After having completed the tour, both 
surveys were handed to all participants. After collecting the completed surveys 
from the participants, the findings were analyzed so that they allow a comparison 
between both user groups. Moreover, based on the subsequent analysis, outliers in 
the responses were identified, and questions were prepared for holding qualitative 
interviews with selected participants for assisting the authors in gaining a better 
understanding about the responses. 
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4.2 Making Visible the Invisible—Augusta Raurica AR App 


The location chosen for carrying out the case study is the archaeological site of 
Augusta Raurica located in Kaiseraugst, Switzerland. The AR-enabled app aims to 
recreate parts of the archaeological site of Augusta Raurica by providing a look into 
what life might have looked like during Roman times. In addition, the app provides 
information about publicly inaccessible artifacts discovered at the site (““APP— 
Making visible the invisible”, n.d.). The AR app is divided into four sections 
focusing on different visitor areas. 

The app supports the visitor to find the locations presented in different sections 
with the help of an online map. The visitor receives information on a physical 
information panel (see Fig. 1). Within the app, the visitor is guided through extra 
information and an audio guide, as well as access to the AR mode. A 3D model will 
be superimposed onto the information panel. Figure 2 shows the AR experience of 
the section “The Baths”. 

For the case study, the app version 1.0.1 was used. Both Android and iPhone 
variants were tested. No difference in experience and usage was noted between the 
two fabrics. 


Fig. 1 Information panel with visitors 
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Fig. 2 Screenshot of the AR mode, section “The Baths” 


5 Results 


5.1 Characteristics of Study Participants 


The case study was conducted on-site at Augusta Raurica in Kaiseraugst, 
Switzerland between January 15 and January 19, 2018, with 16 participants (9 
female/7 male). The participants had various professional backgrounds and were 
aged between 25 and 67 years old. Ten participants were classified as digital native 
and the remaining six as digital immigrants. Table 1 shows the results of the initial 
survey including the classification. 

Digital immigrants were of a higher age with an average of 55 years. However, 
it is important to state that age is not a reliable variable to categorize the participants 
into the respective groups. One participant, who was 63 years old, answered all 


Table 1 Characteristics of user groups 


Digital natives Digital immigrants Overall 
Total participants (female/male) 10 (3/7) 6 (6/0) 16 (9/7) 
Mean age 34 55 42 
Mean number of ICT devices 5.90 4.83 5.50 
Multitasking (%) 100 0 63 
Access to technology (%) 100 50 81 
Confidence in use (%) 100 67 88 
Internet first for information (%) 100 83 94 
Internet for learning (%) 100 67 88 
Internet for school/work (%) 100 33 75 
Good or excellent skills (%) 100 0 63 
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questions positively, and therefore was categorized as a digital native. Digital 
natives had typically more ICT devices in their households. None of the digital 
immigrants said that they were multitasking when using different technologies. 


5.2 Results of the Case Study 


When we observed case study results, digital immigrants rated almost all questions 
higher than digital natives. There were only two exceptions: QI: “I was able to 
learn new things by using the AR app”, Q6: “I found the AR app pleasant and 
user-friendly”. The biggest difference in rating was for Q1O: “I felt like being back 
in time”. While digital immigrants rated a mean score of 4.0, digital natives only 
rated 2.2. The detailed results are shown in Table 2. 

By combining the 12 Questions into the four realms, it is possible to obtain a 
clearer overview. Digital immigrants have rated all four realms slightly better with a 
difference of 0.3-0.6 points on the Likert scale. The best rating by both user groups 
was the education and the worst was escapist realm. Table 3 shows this analysis. 


Table 2 Likert scale rating of the survey 


Question* Digital natives | Digital immigrants | Mean total 
Ql. Learning new thing 3.4 3.2 3.3 
Q2. Made curious and wanted to explore | 3.4 4.5 3.8 
Q3. I enjoyed the learning 3.1 3.2 3.1 
Q4. Visual appearance attractive 2.6 3.5 29 
Q5. Level of detail 2.3 3.2 2.6 
Q6. Pleasant and user-friendly 3.7 353 3.6 
Q7. Could focus 3.5 4.5 3:9 
Q8. Very entertained 2.5 3.2 2.8 
Q9. Was fun 3.7 3.8 3.8 
Q10. Felt back in time 2.2 4.0 2.9 
Q11. Felt like someone else 1.9 1,5 1.8 
Q12. Was completely detached 1.6 2.0 1.8 
Mean total score 33.2 38.8 36.0 


“Questions have been shortened for visibility 


Table 3 User groups rating Digital Digital Mean 

in the experience economy aatives immigrants 

ane Education 3.3 3.6 3.4 
Aesthetic 2.9 3.3 3.0 
Entertainment | 3.2 3.8 3.5 
Escapist 1.9 2.5 2.1 
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6 Interviews 


The qualitative data helped the authors to get a holistic or general overview of the 
differences, whereas the four qualitative interviews held with six participants, two 
digital natives, and four digital immigrants, helped to gain a more detailed under- 
standing about the differences in user experience. 

The paper’s objective of assessing effectiveness in the AR experience in Augusta 
Raurica between the different user groups was successfully met. The results from 
the performed case study revealed that there are expectation differences in the AR 
experience between the user groups. Overall, digital immigrants were more engaged 
by the technology than digital natives. Moreover, during the interview with a digital 
native, it became evident that they had high expectations about the AR Installation 
and the app. The actual capabilities and functionalities of the AR application did not 
match the expectation, as can be derived from the statement below: 


The idea of the app was good in my opinion, and I was very excited before using it. 
However, the moment I started using the AR app I did not like it at all. The app itself was 
complicated, the visuals were nothing special and did not even open correctly on my phone. 
[...] I was expecting much more from the app and to be more interactive. If they don’t 
improve the AR, the app will not find any admirers. I was totally disappointed. (Digital 
Native, 1) 


Moreover, the same attitude can be observed by the statement below from 
another digital native: 


Initially I was quite excited about having to try the AR app at the Augusta Raurica, as the 
idea itself seemed very innovative and intriguing to me. Nonetheless, I have to say that after 
the whole experience I was quite disappointed. I expected a lot more from the app in terms 
of the augmented reality technology. It did not add any value to what you could already see 
in the picture at the sight. (Digital Native, 2) 


Another aspect regarding user-friendliness met the literature review findings. 
Stfelak et al. (2016) found out that the perceived ease of using AR technologies 
differs between older and younger participants. The case study came to the same 
conclusion if we refer to the results of Q6 of Table 2. This is also supported by the 
following statement from a digital native when asked why the young respondent 
gave a good rating to Q6: “I found the AR app pleasant and user-friendly”: 


Well, this is because the app was easy to install and navigate through. The design was 
appealing, and nothing was too messy, so it was quite easy to find what you were looking 
for. (Digital Native, 3) 


Contrary, as part of the qualitative interview an older digital immigrant, when 
asked why a bad rating is assigned to the same question: 


Yes, it confused me. If I had not been looked after by you (referring to the instructor), I 
would not have tried it. I would have given up quickly, it would be difficult. (Digital 
Immigrant, 1) 
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This is in line with Wu et al. (2013), who state that processing information 
through AR can lead to cognitive overload and confusion. Furthermore, Yovcheva 
et al. (2012) identify the inability to effectively compete against traditional means of 
information as a prevailing challenge for AR in the tourism industry. This is 
confirmed by the statement from a digital immigrant below: 


Id rather read something, look at something, but don’t like to deal with technology. Access 
to technology is not my topic. I prefer other ways. (Digital Immigrant, 2) 


Observations showed, that digital immigrants found it hard to install the app and 
start using it. To overcome this issue, digital immigrants suggested to provide 
ready-to-use systems, for example, pre-installed tablets or VR glasses, which would 
be rentable on-site. They would be willing to pay for this kind of service: 


The idea to use VR glasses, which display more details and has more information included, 
would make the installation much more interesting. You would need to pay something for 
such a service, but that is fine. (Digital Immigrant, 3) 


Furthermore, Moorhouse et al. (2017) argue that AR can improve the learning 
process at cultural heritage sites. Participants of the case study were aware of the 
capabilities of AR to enhance the learning process judging by the following 
statement from a digital immigrant: 


Historical backgrounds can be conveyed in a playful way without having to learn. If the 
knowledge comes with pictures, it would be much more interesting to learn. That would be 
less dry for learning: Learning without being aware of it. (Digital Immigrant, 3) 


7 Conclusion 


As already discusses thoroughly throughout this paper, one of the most promising 
technologies that have the potential to support tourist destinations is AR. Although 
the technology has been around for quite some time, we have only recently wit- 
nessed the development and implementation of use cases in the tourism industry. 
Given its infancy stage, the research community and tourist destinations have only 
now Started to conceptualize and understand how the technology can deliver actual 
benefits to the destinations implementing such technologies and to the customers 
using them. 

We looked closely to the AR experience of digital natives and digital immigrants 
with Augusta Raurica’s app. The data from the surveys showed that digital natives 
rated their experience in all four experience economy realms slightly worse. 
Interviews revealed, that digital natives had higher expectations of AR than digital 
immigrants. Deriving from this study, we assess that the tourist destinations with 
AR experiences should target both user groups and aim to fulfil the expectations by 
providing richer content and user-friendly applications. 
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As mentioned in the introduction chapter, our motivation is to assist the research 
community and tourist destinations in better understanding the prevailing differ- 
ences between both user groups, in order to develop tailored and more engaging AR 
experiences and AR content for them. There are clear limitations that we are aware 
of and should be improved in further investigations; small sample size, and the 
single case study site making this study not representative of AR installations in the 
tourism industry in general. Furthermore, the AR installation at Augusta Raurica 
can be considered as a small project and a more developed AR installation might 
have led to different results. Our opinion is that a bigger sample size and performing 
case studies at several state-of-the-art AR installations in tourist destinations could 
lead to better and more detailed results. Subsequently, applying a combination 
between the experience economy and another framework might deliver additional 
input that this study was not able to cover. 
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Abstract Cultural tourism has been identified as an important economic and social 
contributor worldwide. Main drivers have been linked to an increasing desire for 
cultural awareness, meaning-creation and learning. An increasing body of research 
explores the application of VR and AR in this context. While previous studies 
outline VR and AR as promising technologies to positively influence the visitor 
experience, these typically do not focus on how such technologies should be built to 
suit the context or add value to tourists. This study investigates elements affecting 
the tourist experience in the cultural tourism context from a theoretical perspective 
by discussing the impact of VR and AR technology on the visitor’s learning 
experience. It offers contributions in the area of cultural tourism and consumer 
psychology, discussing tourist sites mediated by engaging technologies to enhance 
the visitor experience. Further research is highlighted in the area of VR and AR 
development through purpose-driven design. 
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1 Introduction 


Cultural tourism has long been identified as an important economic and social 
contributor in Europe and globally (Richards, 1996), developing from a niche 
market to a key driver of tourism in a number of destinations. According to 
McKercher, Wong, and Lau (2006), this development has been largely triggered by 
tourists seeking cultural awareness in form of meaning-creation and learning. As a 
result, numerous cultural tourist attractions have attempted to differentiate them- 
selves by exploring opportunities to offer visitors an enhanced experience on site to 
amplify tourists’ enjoyment. Self-motivated and self-guided learning was identified 
as one of the key motivators of visitors to engage with cultural tourism products 
(Ismagilova, Safiullin, & Gafurov, 2015), suggesting a number of use cases to 
support tourists along the visitor journey. Self-motivated and self-guided learning 
have made noteworthy contributions in form of audio guides as well as more recent 
developments such as mobile applications that are able to present information and 
storytelling to the individual’s pace and interest. However, as technology opens 
new opportunities to reshape the visitor experience, a number of studies have 
started to investigate the benefits of augmented (AR) and virtual reality 
(VR) applications in the cultural tourism context (Chung, Lee, Kim, & Koo, 2017; 
Jung, Chung, & Leue, 2015; Raptis, Fidas, & Avouris, 2018). 

Offering an added layer of virtual enhancement, AR and VR have been posi- 
tioned as a promising tool in the cultural tourism context to enhance the tourist 
experience (Jung, tom Dieck, Lee, & Chung, 2016). However, previous studies lack 
discussion on how AR and VR should be implemented for mass adoption and 
return on investment. As previous studies were exploratory, it is questionable 
whether a recommendation to invest in AR and VR is practical from an industry 
perspective. Because research outcomes have depended on prototype applications 
and demos, in-depth investigations of user interaction and impact on the visitor 
experience have been lacking. Thus, we lack the full picture on how AR and VR 
applications impact the visitor experience as a whole. A key question remains 
whether such technologies would help visitors to connect with cultural heritage or 
degrade the experience due to the added digital layer between the tourist and the 
cultural object, or technological challenges that prevent the unobtrusive interaction. 
Such developments need to be carefully designed in order to be perceived as 
meaningful and desirable for tourists. 

This study aims to conceptually outline factors that shape the visitor experience 
in the cultural tourism context with a focus on the visitor’s learning experience as 
key motivator for visiting cultural tourism destinations. Furthermore, the paper 
discusses how AR and VR technology development should be approached to 
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ultimately enhance the overall visitor experience, linking consumer behaviour and 
psychological perspectives with the cultural tourism context in an attempt to bridge 
the two research areas with purpose-driven AR and VR development. 


2 Literature Review 


2.1 Cultural Tourism 


While cultural tourism has often been difficult to define due to the complex nature 
of the meaning of ‘culture’ (McKercher et al., 2006; Richards, 1996), Silberberg 
(1995) characterized cultural tourism as ‘visits by persons from outside the host 
community motivated wholly or in part by interest in the historical, artistic, sci- 
entific or lifestyle/heritage offerings of a community, region, group or institution’. 
Cultural tourism, which was once considered as a niche market, has developed into 
a conventional building block for contemporary tourism and became a key driver 
for many tourists to travel. Thus, cultural tourism turned into an important eco- 
nomic and social contributor in Europe and globally (Richards, 1996) offering 
diverse products and services to a broad target audience. 

Visitors’ self-motivation of understanding and meaning making for cultural 
exposure in the context of foreign and own history motivates millions of tourists 
yearly to engage in cultural sites and destinations. Travelers created an appetite for 
authentic cultural experiences in heritage, ethnicity, cuisine, crafts, arts, and music, 
continues to expand. Cultural tourists are tourists who have interest in visiting 
heritage or cultural sites. According to different tourism studies (Kerstetter, Confer, 
& Graefe, 2001; Silberberg, 1995), they are believed to spend more than the 
average tourist, be highly educated, have a higher disposable income, be older and 
stay longer at a destination. However, culturally-motivated visits range from pur- 
poseful to incidental cultural travel. McKercher et al. (2006) distinguish between 
five cultural tourist categories: (1) Purposeful cultural tourists whose primary 
motivation for visiting a destination is to gain a deep cultural experiences; 
(2) sightseeing cultural tourists whose experiences are less deep but still primarily 
driven by culture; (3) serendipitous cultural tourists who do not travel for cultural 
reasons in the first place, but happen to have a deep cultural experience; (4) casual 
cultural tourists who do not have a particular motive and a shallow experience and 
finally (5) incidental cultural tourists which have no motive whatsoever to travel for 
culture and have very shallow experiences. 

While Liu (2014) argues that the growing cultural tourism segment can be 
attributed to a growing demand for travel driven by economic growth, Falk, 
Ballantyne, Packer, and Benckendorff (2012) and Ismagilova et al. (2015) suggest 
that cultural tourism has often been linked to an increasing desire for cultural 
awareness and learning. Altunel and Erkut (2015) agree that learning, enjoyment 
and escape are the main factors determining the visitor experience in heritage 
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destinations. In this sense, it is important to understand the whole tourism expe- 
rience when visiting a cultural destination—what are the needs and motivations of a 
visitor and can culture, meaning making and learning me made more accessible? 

Recently, experiential and participative tourism activities stimulated by arts, 
authentic artefacts, local festivals and cultural attractions allow tourists to engage in 
and witness extraordinary experiences (Rojek, 2002). Nevertheless, cultural visitors 
primarily prefer to see historic sites, buildings and monuments where they are 
seeking to encounter historic places (Hall & Zeppel, 1990). According to Brida, 
Dalle Nogare, and Scuderi (2016), most museum visitors are searching for a ‘light 
consumption’ experience. Thus, people are not particularly interested in culture 
outside their travel experience. Serendipitous and casual cultural visitors find it 
difficult to connect to presented cultural artefacts. Some tourism scholars (Chang, 
Backman, & Huang, 2014) propose a more creative and engaging form of tourism 
by integrating tourists in an active, long-lasting form of experience. 


2.2 Visitor Experience 


Grounded in early psychological research (see Jantzen, 2013 for an extensive 
review), experience has been an essential object of study in tourism research (e.g. 
Arnould & Price, 1993; Beedie & Hudson, 2003; Sims, 2009; Tan, Kung, & Luh, 
2013; Tsaur, Yen, & Hsiao, 2013; Zajchowski, Schwab, & Dustin, 2016; for 
review, see Scott & Le, 2017). The reason for this is twofold. First, the experience 
of the (cultural) tourist is of essential importance to the cultural tourism industry, as 
experience is the core economic offering in tourism, and adds substantial economic 
value (Pine & Gilmore, 1998). Second, the tourist experience has been of interest to 
academics as it is the main pull factor for tourism destinations, heritage sites and 
related cultural venues, and additionally because of its demonstrable relationship 
with psychological well-being and quality of life (Fredrickson, 1998, 2000). 


2.2.1 Elements Affecting the Visitor Experience 


The interest in the tourist experience has led academics to search for the core 
elements that define or constitute an experience. As reviewed by Scott and col- 
leagues (Scott & Le, 2017), besides the physical context of the heritage site or 
destination—the stage on which the experience takes place—a number of prime 
candidates have been identified: Attention directs our mental resources to stimuli that 
are perceived as being salient. Involvement refers to a person’s level of interest and 
personal relevance in relation to the staged offerings at a site or destination; 
Engagement is a complex construct involving multiple mental processes, all related 
to the feeling of being ‘in the moment’; Immersion is the sensation of being sur- 
rounded by a completely different reality, and is most prominently studied in the 
context of gaming and virtual reality (Ermi & Mayra, 2005). Finally, there is the 
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related notion of cognitive absorption, conceptually close to flow (Csikszentmihalyi, 
1990), in which five dimensions are distinguished: temporal dissociation, attentional 
focus, increased enjoyment, personal control and curiosity. Besides these 
well-established constituent elements of tourist experiences, more recently there has 
been a growing awareness of the importance of emotions in shaping tourist expe- 
riences and in making them memorable (Li, Scott, & Walters, 2014; Moyle, Moyle, 
Bec, & Scott, 2017; Skavronskaya et al., 2017). Hooper-Greenhill et al. (2003) 
among others investigated factors that increase the retention of information and 
enhance learning in the cultural tourism context due to emotional connectedness. 
According to Bond (2014), active visitor engagement was identified as key driver of 
improved information retention. 


2.2.2 Visitor Engagement for Enhanced Learning Experience 


According to Hooper-Greenhill et al. (2003), learning was argued to influence the 
development of attitudes and values, while emotions positively influenced the 
desire to acquire knowledge. This view closely follows Kolb’s definition of 
learning, as “the process whereby knowledge is created through the transformation 
of experience” (Kolb, 1984, p. 38) and the suggested attachment of the learner’s 
emotions. While learning is often seen as a lifelong process, Minocha, Tudor, and 
Tilling (2017) suggest that the learning process entails reflective observation as well 
as active experimentation, both of which affect the overall learning experience and 
are Closely linked to the learner’s emotions. While the Experiential Cycle (Kolb, 
1984) is a useful tool to study the implementation of technology to enhance the 
learning experience, it also identifies a need to clearly understand the factors that 
shape the learner’s experience. Ultimately, understanding how emotions affect the 
concrete experience and abstract conceptualization of the learning process is vital to 
attain a better grasp of what the learning experience entails. 

Implementation of technology to support the learning process was recognized as 
an opportunity to redefine innovative learning possibilities. However, Bond (2014) 
argued that implementation alone was not sufficient to make a measurable impact 
on the learning experience of users. Instead, user interaction had to be carefully 
designed to encourage engagement of the learner with the provided content and 
result in an enhanced learning outcome. Stewart (2014) similarly suggested that 
technology should support concentration and motivation of learners to achieve the 
desired result. Naturally, development of emerging technologies such as AR and 
VR requires in-depth investigation how these can enhance the learning experience 
in a cultural tourism context. 
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2.3 AR and VR 


AR and VR have received a lot of attention since being introduced in the consumer 
market through devices such as Oculus Rift, Google Cardboard or Magic Leap 
Lightwear. However, according to Bonetti, Warnaby, and Quinn (2018), AR as well 
as VR have already been implemented in a number of industry contexts. While both 
types of technology are often packaged into the frame of ‘mixed reality’, it needs to 
be recognized that each has its unique approach to generating a virtual enhancement 
of reality and deserve to be considered individually for purposeful development and 
implementation. AR was defined as the overlay of computerized information that is 
projected into the view of the user through devices such as smartphones, tablet 
computers and wearable devices such as AR glasses (Rauschnabel & Ro, 2016). In 
this form, AR can be categorized into two key pillars, marker-based and GPS-based 
AR. While GPS-based augmentation of the real environment seems to be the logical 
method of AR implementation for tourism purposes, it was argued to lack sufficient 
accuracy as well as processing power of current devices to project meaningful AR 
overlays to enhance the tourist experience (Gherghina, Olteanu, & Tapus, 2013). 
On the other hand, marker-based AR enhancements are triggered through ‘markers’ 
that bind virtual content to specific objects or pictures and were therefore regarded 
the more accessible form of AR enhancements. Apart from mobile-based AR 
applications, site-based AR was mentioned as a third form of AR, which uses fixed 
installations at certain locations such as theme parks and retail outlets, enabling 
virtual augmentations for on-site users (Williams & Mascioni, 2017). In contrast to 
AR, VR uses a computer-generated environment to completely immerse the user 
into a virtual world (Tussyadiah, Wang, Jung, & tom Dieck, 2018). VR has 
received more attention due to the exploding amount of demos and applications, 
both in form of CG animated as well as 360-degree virtual environments particu- 
larly in the gaming and entertainment sector. However, it has made little impact in 
the consumer market relative to expectations and predictions (Abrash, 2016), 
despite the increasing amount of use cases in theme parks or other tourist desti- 
nations. Underlying reasons might be the previously largely limited accessibility to 
consumers due to the need of using a set of VR-glasses or VR-enabling headset 
such as Samsung Gear VR or Google Cardboard using high-end smartphones, or 
the currently limited value that VR can provide compared to the required financial 
investment. A number of cultural tourism providers have attempted to implement 
AR and VR in their context to enhance the visitor experience and attract more 
tourists. 
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2.4 AR, VR Use Cases and Prior Studies in Cultural 
Tourism 


Many cultural tourism sites such as art galleries, museum or cultural heritage sites 
have discovered AR and VR in the past few years. They have enhanced their visitor 
experiences with innovations ranging from virtual enhancements to re-live histor- 
ical sites and events, engage with content in museums, or to visit remote destina- 
tions in virtual environments. Whereas most AR/VR experiences start off as a 
research or pilot project (Fino, Martin-Gutiérrez, Fernandez, & Davara, 2013; Fritz, 
Susperregui, & Linaza, 2005; Han, Jung, & Gibson, 2013), some have recently 
expanded and commercialized. Cultural heritage sites and destinations follow dif- 
ferent strategies in implementing AR/VR. 

A recent study of Marasco, Buonincontri, van Niekerk, Orlowski, and Okumus 
(2018) examined the potential of VR to increase destination competitiveness. The 
study revealed a positive and significant effect of visual appeal (PVA) of VR and 
emotional involvement (EI) on tourists’ behaviour visiting a cultural heritages site. 
Appealing visuals and emotional triggers in VR applications allegedly increase 
likelihood of visiting cultural sites, as AR/VR often portrays an optimal virtual 
representation of the real experience. AR and VR can also increase cultural tourism 
accessibility. Cultural sites welcome a more diverse target group, with different 
interests. To stay attractive for visitors, new technologies are often key for visitor 
engagement. A recent study from Puyuelo, Higén, Merino, and Contero (2013) 
analysed AR as a tool to increase accessibility to architectural and cultural mon- 
ument sites. The AR application supported the understanding of a UNESCO World 
Heritage location by letting users identify and visualise 3D models. The experience 
was positively evaluated, reporting a more engaged experience in terms of aesthetic 
and figurative appeal, enjoyment, and interactivity. 

Industry is following these research pioneers, implementing VR on a large scale 
by launching these technologies mostly as interactive storytelling platforms 
engaging visitors in tourist attractions or urban destinations. Thirty-five major art 
museums in France cooperate in the project eMotion to animate art exhibitions and 
let the visitor travel around the world. Animated characters come to life in a 
symbiosis of photo, art, and digital animation to tell stories and let the visitor 
explore the virtual world (De Paola, 2018). Commercial projects often aim at 
engaging potential visitors in the pre-travel phase to trigger their interest. 
Microsoft’s HoloMaps and HoloTour, for instance, uses 360-degree video content 
and spatial sound to encourage the user to move around the CG-augmented places 
such as Machu Picchu or the Colosseum in Rome without traveling to the actual 
location (Microsoft, 2017). However, as the HoloLens AR headset, which delivers 
these experiences, might not be yet affordable for mainstream tourism applications, 
other commercial projects using VR make use of cheaper headsets such as Google 
Daydream View headset or Cardboard. The Discovery TRVLR application from 
Google and Discovery Communications try to address a bigger audience with their 
VR experiences. The project aims to make remote locations more accessible by 
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inviting the virtual traveller to follow a local host in spectacular tour guides 
(Discovery VR, 2018). Other early adopters, largely developing VR experiences on 
the HTC Vive, have access to much of the VR travel content, ranging from a Grant 
Canyon CR Experience to Stonehenge VR Sandbox (Steam, 2017a, 2017b). 


3 Proposed Framework 


Based on the reviewed literature, we propose the following conceptual framework. 
In this framework, the final aim is defined in the cultural tourism context as the 
overall learning experience of the visitor. The framework adapts the Experiential 
Cycle (Kolb, 1984) to define the effect of AR and VR implementations on the 
visitor’s learning experience. In this regard, the concrete experience, which was 
defined by Kolb and Kolb (2005) by sensory and post-sensory experience of the 
visitor, is linked to the visitor’s emotional attachment of the learning experience. 
Thus, emotions are needed to solidify the impact on the overall learning experience 
and therefore should be clearly understood to avoid creating negative emotions in 
the process, which influences the overall experience. Furthermore, we propose that 
active experimentation is triggered by the degree of visitor engagement with the 
tourism product. The higher the visitor engagement, the higher the impact on the 
learning experience by means of increased active experimentation. AR and VR 
implementation has therefore a high potential to influence the degree of visitor 
engagement through the interactive user experience the technology can provide. 
Furthermore, AR could potentially produce means of enhancing reflective obser- 
vation through virtual enhancements. As abstract conceptualization in the 
Experiential Cycle (Kolb, 1984) was regarded a process that takes place within the 
individual, it is not further discussed in this paper. Nonetheless, to develop and 
implement AR and VR technology meaningfully in the cultural tourism context, it 
is vital to understand the benefit this technology will provide for the end-user. In the 
case of employing AR and VR in the cultural tourism context, it is therefore 
imperative to understand what the underlying tourist motivations to visit the des- 
tination entail. Potential applications need to be developed to support tourists’ 
motivations and should not be developed separately from the overall experience. 
The need to understand how emotions affect the learning experience and the 
resulting overall tourist experience in the cultural tourism context is often over- 
looked, despite being a crucial element in the learning process (Fig. 1). 


4 Discussion 


This paper aimed to investigate elements affecting visitor experience in the cultural 
tourism context by discussing the impact of VR and AR technology and the resulting 
paradoxical effects on the overall experience in the cultural tourism context. In this 
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Fig. 1 Conceptual framework for AR/VR impact on the learning experience of cultural tourism 
visitors 


discussion we reflect on five elements: (1) the importance of customer-centric 
design; (2) the objective of technology implementation; (3) technological issues with 
AR/VR implementation; (4) effects of AR/VR on the visitor experience and (5) the 
increasing use of EEG and physiology in measuring experiences. 


4.1 The Importance of Consumer-Centric Design of Visitor 
Experiences 


As pointed out previously in this paper, tourists are increasingly seeking authentic 
and meaningful visitor experiences. Although the degree of sought meaningfulness 
and authenticity may differ between tourist groups (McKercher et al., 2006), it 
seems that in many cases the authenticity and meaningfulness are more in the eye of 
the beholder, the visitor. As Wang (2000) pointed out, many visitors are seeking a 
specific form of authenticity in the locations they visit, namely existential 
authenticity. What this means is that not so much the objective authenticity of the 
touristic object matters, but the way in which it helps visitors to create their own 
idiosyncratic, meaningful authentic experience. This means that in presenting the 
touristic object, one should carefully consider how this object creates meaning for 
the visitor, how it connects to his/her values and enables the visitor to create his/her 
own version of the experience. This is where customer-centric design becomes 
highly relevant, and in particular so-called user empathy—finding out what truly 
matters to the visitor at the level that the visitor may not even be consciously aware 
(Kouprie & Visser, 2009). Having this deeper level of understanding allows 
AR/VR designers to build emotionally engaging layers to enhance the experience of 
cultural tourism sites. 
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4.2 Objective of Technology Implementation—Need 
for Meaningful Design of Emerging Consumer 
Technologies Such as AR/VR 


Technology has continuously enhanced human life, making processes faster, more 
effective, convenient and accessible. However, when looking at early stages of 
consumer technologies until mass adoption, a common cycle can be observed time 
and time again (Weaver, Jansen, Van Grootveld, Van Spiegel, & Vergragt, 2017). 
While early studies are often focussed on the capabilities of an emerging tech- 
nology to get a full grasp on how the technology performs, later studies tend to 
change the focus to take a larger perspective on the potential value emerging 
technology can provide. Considering AR and VR research, it is time we investigate 
how the technology will benefit consumers, industry and other stakeholders, and 
think about specific value propositions that can be realised through meaningful 
design of AR and VR applications. While firms are still often responsible to design 
and stage consumer experiences in meaningful ways through proper understanding 
of customer needs and wants, a shift towards value co-creation by peers is 
becoming more evident. Particularly with interactive consumer technologies such 
as AR and VR, we propose in the conceptual framework that visitor engagement 
plays a crucial role in the aim of influencing the cultural tourist experience. 
Prebensen (2013) supported this view suggesting that customers should be part of 
the value creating process in order to create meaningful experiences for themselves. 
However, the value that AR and VR are promising to provide needs to be clearly 
understood and relevant for the tourists’ context to encourage use of the application 
and ultimately influencing the cultural tourist experience. 

While a number of studies are highlighting the potential that AR and VR can 
provide in the cultural tourism context, it is not clear at what stage in the visitor 
journey this technology is indeed sought after by visitors and what the economic 
and non-economic benefits entail for other stakeholders. A number of papers have 
highlighted the technological challenges that are still evident with AR and VR 
technologies (Han, tom Dieck, & Jung, 2018; tom Dieck & Jung, 2018) and will 
therefore not be explicitly indicated here. However, it needs to be understood that 
technological challenges such as inconsistent interaction are not only challenges for 
user interaction, but detrimental for the tourist experience. In an industry that 
promotes itself as dealing with ‘experiences’, a small glitch in a visitor application 
could potentially have a much higher cost of damage than the understandably 
underwhelming AR or VR experience. In order to understand how and where AR 
and VR will influence the tourist experience, the visitor journey needs to be fully 
understood. Therefore, we propose that contextual information will play a key role 
in defining and designing the added value of AR and VR enhancements. 
Comparing AR and VR use cases in tourism and retail, it can be observed that two 
rather different stages of the customer journey are tackled. While AR and VR 
studies in tourism often explore how the visitor experience can be enhanced at the 
tourist site (Chung et al., 2017; Jung et al., 2016), studies in the retail industry 
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largely explore the use of AR and VR in the pre-purchase stage, more specifically in 
the product selection process (Bonetti et al., 2018). Evidently, studies in tourism are 
focused on the ‘tourist experience’, however, we should keep in mind that the 
experience is not limited to the activities and engagement on-site, but have a much 
wider scope before and after that is able to influence the overall perception and 
memories of visitors. Furthermore, it seems rather contradicting that AR and VR 
implementation is studied on-site at a time when tourists are looking to engage with 
the destination or attraction. Considering the internal motivation of visitors to make 
the effort and travel to specific tourist sites to learn, be inspired and get emotionally 
attached (Falk et al., 2012), it is questionable whether implementing an application 
to be interacted by means of a device such as smartphones or headsets is the logical 
solution. Arguably, this is creating an additional barrier between tourists and 
tourism product, which potentially prevents the establishment of a deeper con- 
nection and is rather detrimental to the tourists’ internal motivation. This of course 
does not propose that AR and VR implementation should be avoided on-site. 
However, it is crucial to understand and consider tourist motivations such as the 
intention to learn and the value that such technologies can provide along other 
touchpoints of the visitor journey, in order to create meaningful applications that 
will ultimately enhance and not deter the cultural tourist experience. To measure 
how the tourist experience is actually affected, we propose the use of EEG and 
physiology as complementing methods to get a clearer indication on what is 
actually happening at the time of experience consumption when interacting with 
technology such as AR and VR that supposedly enhance the visitor experience. 


4.3 Increasing Use of EEG and Physiology for Measuring 
Experiences in Addition to Reflective Indications 


As discussed in the section on visitor experience, there has been a growing 
awareness that emotions play an essential role in shaping the tourist experience, and 
in making experiences meaningful and memorable (Li et al., 2014; Moyle et al., 
2017; Skavronskaya et al., 2017). This in turn has led academics to consider which 
experience measurement tools would be most effective in capturing the emotional 
dimension of experience (Li et al., 2014). To date the dominant research method- 
ology has been to rely on post-experience self-reports in the form of questionnaires 
or interviews. However, one may question whether relying exclusively on these 
traditional research techniques constitutes the optimal research methodology for 
measuring the emotions that create memorable experiences. It has been argued (see 
Larsen & Fredrickson, 1999 for extensive discussion) that self-reports inherently 
fail to fully capture the essential emotional dynamics of experiences in a sufficiently 
valid manner (Larsen & Fredrickson, 1999). In order to overcome these method- 
ological shortcomings, and to more fully and validly capture the ebb and flow of 
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emotions as an experience unfolds over time, researchers in the field of tourism are 
increasingly using biometric (physiological) measures as well as recordings of brain 
activity. 

Physiological measures such as Skin Conductance Responses (SCR) and Heart 
Rate Variability (HRV) have long been used in psychological research as proxies of 
emotional arousal (Appelhans & Luecken, 2006; Bradley, Miccoli, Escrig, & Lang, 
2008), and can nowadays be reliably recorded with wearable devices (e.g. wrist- 
bands). This technological development allows for reliable emotion measurements 
as tourists are freely walking around and are experiencing a destination or cultural 
heritage site, and therefore has become an accessible and affordable tool for 
scholars in tourism research. Consequently, these tools are increasingly being used 
by researchers in our field. For example, Kim and Fesenmaier (2015) measured the 
SCR of two heritage tour participants in the city of Philadelphia and linked a 
descriptive qualitative analysis of these data to their verbal descriptions of the 
experience. Li et al. (2012) studied HR along with self-reports of emotions while 
tourists were interacting with macaques in a Chinese natural park, and found both 
indicators to reveal positive responses to these interactions. Trondle and colleagues 
conducted a large-scale study on museum visitors (Tschacher et al., 2012) in which 
they continuously measured HR and SCR in more than 500 visitors while their 
exact location was tracked. It enabled them, amongst others, to make ‘emotion 
maps’ of the museum floorplan (Tréndle, Greenwood, Kirchberg, & Tschacher, 
2014), and to identify emotional responses to individual artworks (Tréndle & 
Tschacher, 2012). 

Recordings of electrical brain activity (electroencephalography, or EEG) also 
reliably measure emotional responses (Hajcak, Weinberg, MacNamara, & Foti, 
2012; Harmon-Jones, Gable, & Peterson, 2010). They offer greater precision than 
the physiological measures discussed in the previous paragraph, at the expense of 
only being usable in a laboratory setting. Ongoing work in our research group is 
seeking to validate the use of so-called frontal alpha EEG asymmetry (which is a 
continuous EEG-based measure of positive and negative emotions; Harmon-Jones 
et al., 2010) in experience research. In this project, short VR movies (durations 
ranging from three to 14 min), delivered through Samsung VR Gear equipment, 
were used to engage research participants in an immersive experience. Preliminary 
analyses show that, amongst others, there are significant correlations between 
valence ratings and frontal alpha asymmetry. These findings validate EEG as a tool 
to study, with sub-second resolution, the succession of positive and negative 
emotions during an experiential episode, which bypasses the use of self-reports. 

As said, a major limitation of EEG as a tool for measuring emotions during a 
tourist experience is that it can only be reliably recorded in a lab setting. It is 
precisely here that we see a great potential advantage of combining EEG mea- 
surements with AR/VR technology, as this technology allows for immersing 
tourists and visitors in realistic, ecologically valid experiences while at the same 
time being in a well-controlled laboratory condition. EEG is therefore a potentially 
very useful tool for AR/VR experience design and optimization: it can be used for 
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evaluating the emotional contents of an AR/VR experience, and by systematically 
varying elements of the AR/VR experience and subsequently optimizing the design, 
it allows for truly evidence-based AR/VR experience design. 


5 Conclusion 


The aim of the present paper was to examine the connection between AR/VR and 
the visitor experience of cultural tourism attractions. Cultural tourism institutions 
are increasingly implementing technologies such as AR and VR. Academic 
research on these experiences is in a nascent stage, and is in need of theoretical 
development. Based on the Experiential Cycle of Kolb (1984), we have proposed a 
theoretical model for understanding the visitor experience of AR/VR in the context 
of cultural tourism. This model implies the need for further research into appro- 
priate measurement methodologies of these experiences. Biometric methods such as 
EEG and wearable measurement of peripheral emotion physiology holds particular 
promise herein. Furthermore, additional research is needed to develop existing 
theories of cultural tourism to keep pace with the technological landscape. The AR/ 
VR technologies discussed, the software they use, and the cultural tourism expe- 
riences they can support are becoming increasingly accessible and, therefore, 
increasingly widespread. It is reasonable to predict that AR and VR will soon be 
seen as common dimensions of cultural tourism experiences. It is our urging that 
academic research in cultural tourism should keep pace. 
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on Enriching Visitors City Experience ‘ates 
with Augmented Reality: 

An Exploratory Study 


Natasha Moorhouse, Timothy Jung and M. Claudia tom Dieck 


Abstract The purpose of this exploratory study is to provide initial insights into 
tourism marketers’ perspectives on and intention to adopt Augmented Reality 
(AR) to enhance the visitor experience in urban tourism destinations. Interviews 
were conducted with seven participants from four Destination Marketing 
Organisations (DMOs) in three urban tourism destinations in the UK. Using the- 
matic analysis, the results were mapped to the Technology, Organisation, and 
Environment (TOE) framework. Three sub-themes were identified in accordance to 
each component of the framework, totalling nine themes overall. The findings 
highlighted four perceived opportunities of tourism marketers AR adoption 
including three technological factors: valuable visitor experiences, visitor satis- 
faction, and increased social media presence, and one environment factors: positive 
destination image. Further, the perceived challenges included three organisation 
factors: resource limitation, perceived financial cost, and return on investment, and 
two environment factors: visitor adoption and external concerns. Theoretical 
contributions, practical implications, and recommendations for further research and 
action are presented. 


Keywords Augmented reality - Tourism marketers - Urban tourism destinations - 
Visitor experience 


1 Introduction 


Tourism marketers such as Destination Marketing Organisations (DMOs) are 
challenged with adapting their ways to adhere to the transformation of the designed 
experience offerings into personalised experiences (Volo, 2009). Indeed, the use of 
modern technologies is critical for destinations (Marasco, Buonincontri, van 
Niekerk, Orlowski, & Okumus, 2018), and tourism marketers show a keen interest 
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in new technologies to guide visitors (Baktash, Krey, Nair, & Rauschnabel, 2018). 
Recent advancements in Augmented Reality (AR) has meant that it could 
tremendously advance the tourism landscape (Baktash et al., 2018). In the tourism 
literature, AR has received increasing acknowledgment as a tool to facilitate pos- 
itive visitor experiences in a variety of contexts (e.g. Chung, Lee, Kim, & Koo, 
2017; tom Dieck, Jung, & Rauschnabel, 2018) including urban tourism (e.g. 
Markouzis & Fesakid, 2015). However, studies exploring tourism marketers’ views 
on this topic in a specific urban tourism context are limited. Indeed, this perspective 
is required to gain insights into industry perceptions and intention to adopt this 
innovative technology. Extensive research indicated that barriers to new technology 
adoption at firm-level exist in the internal and external environment, and more 
specifically, adoption is determined by technological, adopter-specific, and envi- 
ronment factors (Cooper & Zmud, 1990; Hafeez, Keoy, & Hanneman, 2006). 
Therefore, this study aims to contribute preliminary research by exploring tourism 
marketers’ views on adopting AR to enrich visitors’ city experience, by identifying 
influential adoption factors existing in both the internal and external environment. 
More specifically, this study has two research objectives: 


(1) To identify the perceived opportunities and challenges in the internal envi- 
ronment influencing tourism marketers’ AR adoption 

(2) To examine the perceived opportunities and challenges in the external envi- 
ronment influencing tourism marketers’ AR adoption. 


This article begins with an overview of how Information and Communication 
Technologies (ICTs) have forced tourism marketers to change their ways to remain 
competitive. This is followed by an exploration of the tourism literature on AR and 
visitor experiences with specific focus on urban tourism destinations. Then, the 
methods and findings and discussion are presented, and finally, conclusions, limi- 
tations, and avenues for future research are offered. 


2 Literature Review 


2.1 Impact of ICTs on Tourism Marketers’ 
Competitive Strategy 


DMOs are responsible for the strategic management and marketing of destinations 
(Martins, Costa, & Pacheco, 2013). They must play an active role in the destina- 
tions competitive position in the industry by gaining increasingly demanding and 
experienced consumers through the intensive use of ICTs (Martins et al., 2013). 
The way people find destination-related information, purchase travel products, and 
experience, communicate and perceive destinations has changed (Buhalis & Jun, 
2011; Neuhofer, Buhalis, & Ladkin, 2012; Xiang & Gretzel, 2010). This is largely 
due to new technological trends and changes in software and infrastructures, as well 
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as human participatory culture involved in virtual environments (Buhalis & Jun, 
2011; Neuhofer et al., 2012; Xiang & Gretzel, 2010). To date, the internet remains 
important in the communication of destination marketing to secure new and 
existing clients (Li, Robinson, & Oriade, 2017). In addition, word-of-mouth 
(WOM) communication, where consumers can converse about their experiences in 
an open channel and customer reviews (Oskarsson & Georgsdottir, 2017) remains 
fundamental. In 2013, Hays, Page, and Buhalis (2013) stated that DMOs are turning 
to social media as a relatively low-cost and global reach marketing tool. Prior to 
this, several studies (Schegg, Liebrich, Scaglione, & Ahmad, 2008; Stankov, Lazic, 
& Dragicevic, 2010; Wang, Quaehee, & Fesenmaier, 2002) suggested those tourism 
organisations that fail to adopt social media will lack a competitive advantage. The 
increase in User Generated Content (UGC) is a result of the internet and mobile 
technologies that provide individuals with unprecedented power to instantly add 
“digital traces” when conducting tasks (Lu & Stepchenkova, 2015). This could 
include reviewing tourism service providers, documenting a travel experience, or 
uploading photos and videos, hence, social media platforms contain a wealth of 
UGC (Lu & Stepchenkova, 2015). It is therefore evident that tourism marketers are 
continuously exploring ICTs and new forms of human interactions with a view to 
offer new opportunities for visitors to engage in the process of co-creating enhanced 
destination experiences in a technology-based environment (Agapito & Lacerda, 
2014). Indeed, digital tourism was previously concerned with the use of digital 
technologies to enhance the visitor experience such as posting recommendations on 
a tourist website (Benyon, Quigley, O’Keefe, & Riva, 2014). However, more 
recently the focus has shifted and is increasingly concerned with mixing of the real 
world with digital content designed to enhance the visitor experience, and with the 
proliferation of smartphones and tablet devices, AR is becoming more widespread 
(Benyon et al., 2014). 


2.2. AR-Enhanced Visitor Experiences 


In the tourism literature, AR-enhanced visitor experiences have received strong 
interest in a variety of contexts including urban heritage tourism (e.g. Han, Jung, & 
Gibson, 2013; tom Dieck & Jung, 2015), nature-based tourism (e.g. Linaza, 
Gutierrez, & Garcia, 2014), cultural heritage tourism (e.g. Chung et al., 2017; 
Cranmer & Jung, 2014; Jung, Chung, & Leue, 2015; Seo, Kim, & Park, 2010), 
museums (e.g. Haugstvedt & Krogstie, 2012; Jung, tom Dieck, Lee, & Chung, 
2016; Moorhouse, tom Dieck, & Jung, 2017), art galleries (e.g. Leue, Jung, & tom 
Dieck, 2015; tom Dieck, Jung, & tom Dieck, 2016), city tours (e.g. Fino, 
Martin-Gutiérrez, Fernandez, & Davara, 2013), travel guide (e.g. Kourouthanassis, 
Boletsis, Bardaki, & Chasanidou, 2014), theme park (e.g. Jung, Chung, & tom 
Dieck, 2015), and festival (e.g. tom Dieck et al., 2018). Studies have indicated that 
AR provides visitors with the opportunity to become more informed on unknown 
surroundings in an enjoyable and interactive manner (Han et al., 2014), by 
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providing interesting and valuable information and therefore enhancing the overall 
touristic experience (tom Dieck & Jung, 2015). In an urban tourism context, AR 
makes visiting cities more efficient and meaningful by supporting the improvement 
of the environment perception by users’ interaction, while the GPS enables the 
location-based retrieval of information and services (Markouzis & Fesakid, 2015). 
Visitors are empowered to enjoy more freedom in searching, viewing, selecting, 
and purchasing products and services (Dadwal & Hassan, 2016). The important 
functions of AR are its resources at the destination, agenda, photo gallery, and the 
possibility of sharing comments (Markouzis & Fesakid, 2015). From a managerial 
perspective, ARs development and operation offers new employment and business 
opportunities (Markouzis & Fesakid, 2015), and its use in tourism will aid in 
designing effective marketing strategies in the future (Daim, Barri, Kamarudin, & 
Zakaria, 2012). However, the use of AR for tourism marketing purposes is fully 
dependent on the access and resource availability (Rahimi, Hassan, & Tekin, 2016). 
Indeed, innovative technologies including AR are forcing tourism marketers to 
adapt their competitive strategy. However, limited studies have investigated their 
perceptions on AR and their intention to adopt this technology. Therefore, this 
study aims to explore the perceived opportunities and challenges in both the internal 
and external environment to demonstrate tourism marketers’ current views on AR 
and adoption intentions of AR. 


2.3. Tourism Marketers’ Adoption of Innovative 
Technologies 


Theories dedicated to IT adoption at firm level have been developed. Most studies 
investigating this perspective are derived from two theories: Diffusion of Innovation 
Theory (DOD (Rogers, 1995) and the Technology, Organisation, and Environment 
(TOE) framework (Chong, Ooi, Lin, & Raman, 2009; Tornatzky & Fleischer, 
1990). The DOI theory relates to how, why, and at what rate new ideas and 
technology spread through culture operating at the individual and firm level 
(Oliveira & Martins, 2011). However, the DOI theory does not consider the 
environment context that other theories such as the TOE framework consider, and 
thus, other theoretical models are considered to be more complete (Oliveira & 
Martins, 2011). The TOE theory identifies aspects of an organisations context that 
influence the process by which it adopts and implements a technological innovation 
including the technological context, organisational context, and environmental 
context. Table | provides an explanation of each context in more detail. 

There is a plethora of research investigating organisational acceptance of new 
technologies using the TOE framework as a theoretical foundation (e.g. Chang, 
Magobe, & Kim, 2015; Gibbs & Kraemer, 2004; Iacovou, Benbasat, & Dexter, 1995; 
Kuan & Chau, 2001; Wang, Li, Li, & Zhang, 2016; Zhu & Kraemer, 2005). For 
example, Iacovou et al. (1995) investigated perceived benefits (technological context), 
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Table 1 TOE framework dimensions 


Context Description 

Technological Describes both the internal and external technologies relevant to the firm 

context (e.g. current practices and equipment internal to the firm, and available 
technologies external to the firm) 

Organisational The descriptive measures about the organization (e.g. scope, size, and 

context managerial structure) 

Environmental The area in which a firm conducts its business (including its industry, 

context competitors, and government interactions) 


Source Oliveira and Martins (2011) 


organisational readiness (organisational context), and external pressures (environment 
context) as drivers of Electronic Data Interchange (EDI). Similarly, Kuan and Chau 
(2001) explored perceived direct benefits (technological context), perceived financial 
cost, perceived technical competence (organisational context), perceived industry 
pressures, and perceived government pressure (environmental context) in the context 
of EDI in small businesses. In the tourism literature, the TOE framework has been 
applied in a variety of contexts such as hotel adoption of reservation systems (Wang 
et al., 2016), e-commerce usage (Chang et al., 2015), mobile technology adoption in 
travel agencies (Lin, 2016), and adoption of electronic customer service management 
(eCRM) systems in hospitality organisations (Racherla & Hu, 2008). Although the 
specific elements identified within the three contexts vary across different studies, the 
TOE framework shows consistent empirical support (Lippert & Govindarajulu, 2006). 
Indeed, there are limited studies applying the TOE framework in the context of tourism 
marketers’ perceptions and intention to adopt AR in urban tourism destinations. 
Therefore, given that this research investigates the perceived opportunities and chal- 
lenges in the internal and external environment surrounding this topic, the TOE 
framework provides a useful theoretical base for this study. 


3 Methods 
3.1 Context of Study 


Exploratory studies provide insights into, and a fuller understanding of, an issue or 
situation, and are well suited to qualitative research methods such as interviews 
(Saunders & Lewis, 2011). Therefore, the present study adopts a qualitative 
interview approach to investigate tourism marketers’ perspectives on and intention 
to adopt AR to enrich visitors’ city experience. A multiple case study strategy was 
employed using three urban tourism destinations in the UK, and interviews were 
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conducted with seven employees from four DMOs in the three selected destina- 
tions. Interview participants included one Commercial Director (P1), four 
Marketing Managers (P2, P4, P6, and P7), one Digital Executive (P3), and one 
Head of Visitor Economy (P5). P1—P7 represents the coding allocated for the 
analysis. Since DMOs are central to the marketing of regional tourism (McCamley, 
Gilmore, & McCartan-Quinn, 2012), and many nations and cities are now funding a 
DMO as the main vehicle to compete and attract visitors to their distinctive place or 
visitor space (Pike & Page, 2014), the selection of a DMO sample seems appro- 
priate. The interviews took place during summer 2017 (lasting approximately 30- 
40 min) and broadly explored the perceived opportunities and challenges in the 
internal environment and the external environment influencing tourism marketers’ 
intention to adopt AR to enhance the visitor experience in this specific context. 


3.2. Data Analysis 


Thematic analysis is a useful method for identifying and reporting patterns and 
themes (Costa, Breda, Pinho, Bakas, & Durao, 2016), and given the exploratory 
nature of this study, this method was perceived to be the most suitable. The analysis 
of the findings where mapped to the three contexts of the TOE framework given 
their relevance to this study investigating the adoption of technologies at firm level. 
Using the three TOE contexts as initial themes, three sub-themes emerged under 
each theme, totalling nine overall, as presented in Table 2. Furthermore, the themes 
are presented in more detail using direct quotations from the data set to support the 
discussion. 


Table 2 Themes and sub-themes 


Themes Technological context 
Perceived opportunities 

Sub-themes Valuable visitor Visitor satisfaction Increased social media 
experiences presence 

Themes Organisational context 


Perceived challenges 


Sub-themes Resource limitation Perceived financial Return on investment 
cost 
Themes Environmental context 
Perceived opportunities Perceived challenge 
Sub-themes _| Positive destination Visitor adoption External concerns 


image 
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4 Findings and Discussion 


4.1 Technological Context 


4.1.1 Valuable Visitor Experiences 


Numerous opportunities for creating more valuable visitor experiences emerged 
such as “AR for navigation or for searching for products nearby attractions” (P2), 
and “for visitors to be guided around the city through AR, such as walking tours ... 
to access offers and deals, and for visiting new places ... especially from the historic 
perspective, there would be lots of opportunities to show what areas used to be like 
... [could see AR working in that way” (P1). In support of this, P4 stated, “AR can 
enhance experiences, whether its increasing interactivity in museums or accessing 
offers”. Further, the potential of AR in creating personalised experiences emerged 
as P1 stated, “it will definitely add value to the visitor experience. If there was a 
targeting mechanism within the application then the visitor will be seeing and 
visiting things that are more in tune with what they like, so to me that equals a more 
satisfied visitor”. However, “the most important is the content ... so that visitors 
feel more fulfilled and satisfied by our destination” (P2). 


4.1.2 Visitor Satisfaction 


Participants confirmed that implementing AR into the visitor experience could 
provide visitors with a more unique experience than they have previously experi- 
enced and allow them to visit attractions they might not have previously considered. 
Therefore, this would increase visitor satisfaction and generate visitor expenditure 
throughout the entire tourism offer (P1, P2, P4, P5, P6, P7). For example, P5 stated, 
“T think AR could change visitor satisfaction because it focuses on enhancing the 
experience once they are here”. In addition, “I hope AR could be something that 
would make the visitor feel they have had a unique experience, that they are getting 
something that not everyone is getting” (P2), and “they would be trying different 
things to eat, going to different parts of the city, seeing new attractions ... it would 
give them chance to find out about new attractions” (P1). “It will certainly add to 
visitor satisfaction ... they would walk away like they had been more immersed in 
the city, like they had been provided with something more, and experienced 
something more in their own eyes” (P6). Overall, “I think there is big potential for 
AR ... it would bring the destination to life” (P6). 


4.1.3 Increased Social Media Presence 


Several participants (P1, P2, P3, P6) noted that implementing AR into the desti- 
nation experience could be a useful way to encourage visitors to share their 
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experiences on social media and enhance the organisation’s social media presence. 
Drawing on first-hand experiences P6 stated, “just from friends and family that have 
been to the destination and used AR, they have really enjoyed it and shared 
experiences on social media and it has been fun for them to do, so I think it has got 
potential ... it is a good way to get the city product out there on social media”. In 
addition, “if AR had a social aspect, that would be a good way to get our brand out 
there because UGC always works best anyway” (P3). Moreover, it is important to 
integrate newly adopted technologies into existing platforms where funding is 
already implemented: “as an organization, we invest a lot of money into online and 
social media so AR could be integrated into that” (P6). Importantly, AR provides 
visitors with “something to show off about ... we value that user-generated, driven 
approach to marketing because we don’t have a huge budget so we rely on small 
things like WOM and social media” (P2). Further, P1 added “if there is something 
visitors can take away with them, to create memories, and sharing them via social 
media, that is undoubtedly a great way to get our brand out there. It works both 
ways—it can work as a negative as well, but generally it is really good”. 


4.2 Organisational Context 


4.2.1 Perceived Financial Cost 


The main concern for all the tourism marketers (P1—P7) when investing in tech- 
nologies such as AR is perceived financial cost and the limited funding that is 
available to tourism organisations such as DMOs. This is evident as P2 stated, “the 
costs are the major challenges ... because the costs are quite high, you have really 
got to evaluate the commercial return that you are going to get from it”. Likewise, 
PS added, “I wonder how these things will be funded ... our recent project costs 
fifteen thousand pounds, which is a lot of money, and we wouldn’t have been able 
to do that without the funding”. P1 pointed out that, “our budgets are different now 
to how they were five years ago. Going back five years we would have EU money 
and funding and since the demise of the regional development agencies etc. we 
have turned ourselves into more of an agency. We are very much self-funding 
projects and campaigns so that budget for the likes of the development projects and 
things, I don’t know where that money would come from”. Nevertheless, P3 stated 
that the organisation has considered investing in AR despite the cost, “although it 
costs quite a lot of money to implement AR applications and features, we were 
considering investing in that area for a recent campaign targeting the international 
market. There is potential definitely”. 
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4.2.2 Resource Limitation 


The findings indicated that resource availability in terms of developing and 
implementing AR technologies could require more time to allocate funding, 
training of employees, and time to gain knowledge and better understand where 
ARs value would be best situated in the urban tourism offer (P1, P3, P5, P6). For 
instance, P3 stated “I do not know how we would utilize AR or what platform we 
would use”, and P5, “I am sure there is potential for AR ... as with anything 
through, we need to identify where it would add value to the visitor experience”. 
Further, “in terms of leisure, I think AR will be there in the future but I don’t know 
how”. In addition, P6 added “I think it would be worth the investment, however, I 
am not aware of the cost ... if it wasn’t massively expensive it would definitely be 
worth it”. Given the perceived high investment costs, P3 added “I don’t know what 
the numbers would be in terms of increasing visitor number ... it would have to be 
a trial run depending on the response we get ... it would be hard to measure ... we 
would rely on [an external organisation] to analyse footfall”. 


4.2.3 Return on Investment 


Assessing the return on investment (ROI) was raised as a key concern for all 
participants (P1—P7). Providing an incentive for visitors to use the application 
would provide tourism marketers with a method to analyse the ROL as P2 stated, “it 
is hard to say whether the benefits would be worth the investment from destination 
marketers over time ... it is about evaluating the commercial return, so there would 
need to be an incentive for visitors to purchase something or access promotions on 
the application to be motivated to get on board”. P1 argued that the “traditionalist 
market” (i.e. older generation) “still like to have printed visitor guides. They won’t 
use their mobile phone to find their way around the city but they all have spare time, 
disposable income. Will they be tech-savvy enough to use something like that? I’m 
not sure”. Nevertheless, P2 noted the potential future sales and engagement 
opportunities that could derive from the initial download of an AR application. For 
example, “AR experiences could motivate people to revisit and if they have 
downloaded the application previously ... it gives us the opportunity to resell them 
something ... it gives us a communication channel of continual engagement” (P2). 


4.3. Environmental Context 


4.3.1 Positive Destination Image 


According to the findings, deploying AR technologies throughout the city’s tourism 
offer has the potential to change the destination image from the perspective of 
visitors, however, this is dependent upon the type of visitor/individual they are and 
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their unique preferences (P4, P5, P6, P7). This is evident as P6 argued, “imple- 
menting AR has the power to change the destination image ... there is such a broad 
range of visitors to the destination and I think it would depend on their prefer- 
ences”. In addition, P4 stated that, “it could potentially change perceptions and the 
desire and interest to visit”, and “I think AR could change the destination image 
from the visitor perception” (P4). Further, several participants identified how 
implementing innovative technologies such as AR could challenge visitors’ pre- 
conceptions of the destination if they are first time visitors. For example, “we 
struggle with the US market because they either have no idea what [the city] is like, 
or they have the wrong perception of it still being an industrial city from years ago. 
We work closely with the media, and influencers to change that perception of the 
global market ... I think AR could really help with that” (P4). 


4.3.2 Visitor Adoption 


Encouraging visitors to adopt AR applications when at the destination could be 
challenging and would be dependent on the perceived benefits and therefore, would 
require an incentive for usage (P1, P2). As P1 stated “another challenge is getting 
people to use the application. I guess that is going to be a generational issue ... It 
might be directed at millennials, but with the cost implications for development and 
the service, would the end price be out of reach for that market? Those are all the 
things that must be considered”. In support of this, P6 stated, “the barrier is 
awareness and getting people to use the AR application once it is available. It needs 
to be made as easy as possible for people to use and understand. I see that as a 
massive challenge. It needs to be quite clear on what the AR application is trying to 
do”. In addition, P2 highlighted that targeted messages through AR could be 
problematic and deter visitor adoption as “I think that is bordering intrusive and 
people could start to shut off from those things. Having said that, a [local organi- 
zation] is starting a WiFi project [in the city] where people sign up and they receive 
push messages about things they are near as they are exploring the city”. 


4.3.3 External Concerns 


Several concerns were raised with regards to concerns in the external environment 
such as visitors’ mobile device capacity (P2), WiFi in the city (P2, P3), usability 
(P2), and user acceptance (P3). For example, P2 stated “things like people’s phones 
being old or out of date and not having the ability to use the application well, they 
also rely on being housed within an application and peoples phone capacities might 
limit this. Also, at the moment the provision of WiFi is pretty bad in the cities, often 
in [the city] 3G is inaccessible so if you want to download something on the go it 
would be quite difficult to do so”. In support of this, “having WiFi in the city that is 
accessible is something we need to consider ... It would definitely set us ahead of 
the crowd and make the city more attractive to the younger crowd” (P3). 
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Nevertheless, P2 explained “overtime destinations will adopt the technologies but at 
the moment it is still early days ... I think there will be a lot of trial and error over 
the next few years”. At the moment, “we don’t feel like we are missing out as a city 
because we don’t have lots of AR experiences, but that might be different in ten 
years’ time” (P5). Having said, given that “AR is mobile enabled, it would be more 
attractive for user acceptance ... the local airport is considering implementing AR 
because they are already using Beacons technology” (P3). 


5 Conclusion 


This exploratory study investigated tourism marketers’ perspectives on imple- 
menting AR to enrich visitors’ city experience by exploring the perceived oppor- 
tunities and challenges of AR adoption in both the internal and external 
environment. By drawing on the TOE framework, it was revealed that the perceived 
opportunities are related to the technological and environmental contexts, while the 
perceived challenges are related to the organisational and environmental contexts. 
AR implementation could generate more valuable visitor experiences and increased 
visitor satisfaction (e.g. Han et al., 2014; tom Dieck & Jung, 2015), thus, increasing 
destination competitiveness (technological context). The significance of UGC in 
tourism marketing and its amplification through social media platforms (e.g. 
Twitter, Facebook, and TripAdvisor) is evident (e.g. Lu & Stepchenkova, 2015), 
and the present study further complements this breadth of research by suggesting 
for AR to be seamlessly integrated with existing platforms to increase the organi- 
sations social media presence and optimize existing channels (technological con- 
text). From the visitor perspective, integrating AR could improve destination image 
by portraying the destination as more innovative (environmental context). However, 
access to and availability of resources (e.g. Rahimi et al., 2016) including funding 
limitations (Hays et al., 2013) are critical factors negatively associated with tourism 
marketers’ AR adoption, and of importance to tourism marketers is evidence of ROI 
(organisational context). Hence, encouraging visitors to download and utilise the 
AR application (environmental context) once it’s been developed is concerning for 
tourism marketers given the financial constraints and concern for wasted resources. 
Finally, technological advancements (e.g. WiFi in the city and visitors’ mobile 
device capability) are required for visitors to take full advantage of AR in the urban 
tourism environment, therefore, indicating tourism marketers’ adoption of AR 
will likely arrive in the near future in accordance to smart tourism initiatives (en- 
vironmental context). 
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6 Theoretical Contribution 


The present exploratory study complements research relating to tourism marketers 
use of innovative ICTs such as AR to enrich the visitor experience. Several studies 
have employed a visitor/user experience perspective when investigating AR and 
visitor experiences (e.g. Han et al., 2014; Markouzis & Fesakid, 2015; tom Dieck & 
Jung, 2015). However, this study adds an alternative perspective to this body of 
literature as such research is limited. More specifically, the findings offer some of 
the perceived opportunities and challenges in both the internal and external envi- 
ronment that tourism marketers could encounter when implementing latest tech- 
nologies such as AR, therefore, contributing theoretically to the tourism literature 
and indicating avenues for further research to expand on this exploratory study. 
Furthermore, this study employs the TOE framework in a largely unexplored 
context, therefore, offering some contribution to TOE research by mapping several 
factors to the three dimensions of the framework that could influence tourism 
marketers’ adoption of AR technologies. However, further research is required in 
order to strengthen this potential contribution and reveal additional emergent 
themes. 


7 Practical Implications 


The findings provide practical implications for tourism professionals, researchers, 
and AR designers and developers. First, tourism professionals could potentially 
benefit from the perceived technological (valuable visitor experiences, visitor sat- 
isfaction, and increased social media presence) and environmental (positive desti- 
nation image) opportunities of AR implementation in this study, which could 
effectively encourage investments in AR. Second, this study provides initial 
exploratory research for tourism researchers on tourism marketers perspectives on 
implementing AR technologies to enhance visitors’ city experience. Third, the 
empirical evidence presented in this study could be beneficial for designers and 
developers as it indicates that tourism marketers require AR applications that are 
cost-effective, simple to implement, and able to seamlessly integrate with existing 
channels (e.g. social media), therefore, requiring limited time (e.g. training 
employees to develop/use) and monetary investment (due to funding constraints). 


8 Limitations 


This study has some limitations which could be mitigated by future research. 
Firstly, the sample size was limited therefore limiting the scope of the analysis. 
Although the sample allowed for reasonable conclusions to be drawn, it cannot be 
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considered representative of the entire population of DMOs in UK-based urban 
tourism destinations. Second, the empirical part explored AR mobile applications 
without providing a physical demonstration and therefore assumed participants 
prior understanding of AR applications. However, participants had varying levels of 
AR knowledge and experience with AR technologies, therefore, experimenting 
with an AR application during qualitative data collection could have allowed for 
more in-depth conversations. Finally, given the qualitative nature of this study, 
future research could employ quantitative approaches and analyses to compliment 
and extend the insights presented within this study. 
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Part IV 
AR & VR in Education 


Creating Virtual Reality in a Business ®) 
and Technology Educational Context oe 


Diana Andone and Mark Frydenberg 


Abstract Virtual reality applications have enabled students to experience the 
world through visually immersive experiences in business and educational contexts. 
This paper shares the results of a collaborative project in which students from 
universities in the United States and Romania work together to study uses for 
virtual reality in a business context, and then create their own VR scenes for a 
selected business or industry. In doing so, students follow a virtual mobility 
learning scenario in which they explore the capabilities of virtual reality in both 
business and technology educational contexts. 


Keywords Virtual reality - Mobile technologies - CoSpaces - Project-based 
learning - Collaborative learning - Online communication 


1 Introduction 


Creating opportunities to learn about Virtual Reality (VR) from business and 
technology perspectives was the driving force for TalkTech 2017, the ninth edition 
of the TalkTech project, a virtual mobility, collaborative learning project between 
students at universities in the United States and Romania. To complete this project, 
students from each school research relevant applications of VR in a chosen business 
or select an industry, visit a local business or location related to that industry, and 
create their own VR scenes to share with their international partners. In this project, 
students used CoSpaces, a VR web-based authoring tool, to create their original VR 
content. The authors argue that the TalkTech project fosters entrepreneurship and 
helps students build their own technology literacy skills. 
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The TalkTech project described in this paper extends ongoing research on stu- 
dents’ attitudes toward AR and VR tools and their applications (Andone & 
Frydenberg, 2017; Frydenberg & Andone, 2017a, 2017b; Li, Li, Zheng, & Zhao, 
2015; Vishwanath, Kam, & Kumar, 2017). In addition, the authors have studied 
how students use information and communication technologies (ICT) such as video 
chat, messaging apps, email, and collaboration tools and their impact on compu- 
tational thinking (Andone & Frydenberg, 2014) and digital literacy (Andone & 
Frydenberg, 2017). 

The research contribution in this paper is a discussion of the following questions 
about student learning related to digital literacy and VR business applications when 
creating original VR artefacts: 


e What is the students’ experience using tools and applications to create original 
VR artefacts? 

e How does the process of creating VR artefacts give students insights into future 
business applications of these technologies? 

e What challenges and opportunities are presented when introducing VR in a 
global collaborative learning project? 


2 Applications of VR in Education 


AR and VR have been referred to as “the fourth wave” in technology platforms 
(Kamenovy, 2017). Following personal computers, the Internet, and mobile devices, 
AR and VR applications now are taking their place in both business and educational 
contexts. Like their predecessors, AR and VR have changed the way we connect 
and interact with the people and the world around us. “VR is an opportunity to ... 
connect with both learners and teachers in a novel and meaningful way” 
(Kolo, 2017, p. 1) as users are able to experience and remember their learning. The 
availability of mobile devices, Google Cardboard and dedicated VR headsets such 
as Oculus Rift and HTC Vive, combined with a growing collection of free apps and 
tools, have made VR a worthwhile educational opportunity. Dalgarno and Lee 
(2010) explore the benefits of 3-dimensional learning environments. They find that 
“there will be more effective real-world application of newly acquired knowledge 
and skills if the learning environment is modelled on the context in which the 
knowledge is expected to be applied” (Dalgarno & Lee, 2010, p. 21). Adoption of 
these tools may enhance or improve the learning process. The introduction of 
Google Cardboard (“Google Cardboard,” 2018) and the ubiquity of smartphones 
make the experience widely accessible. Advantages for introducing VR an edu- 
cational context include relevance, creativity and engagement, interacting with new 
technology, and innovative ways to communicate and share information 
(Greenwald et al., 2017; Kolo, 2017; Reynard, 2017). 

A recent study (Ott & Freina, 2015) shows that virtual reality and immersive VR 
experiences have been used in higher education, mostly in the United States and the 
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United Kingdom, for adults seeking specialized training, or at universities, espe- 
cially in the areas of sciences and medicine. High schools also support VR for 
physics and chemistry simulations. In the K-12 space, virtual reality is one of the 
most popular developments in educational technology, as Google Expeditions and 
Google Earth allow students to visit landmarks around the world without leaving 
their classrooms (Vishwanath et al., 2017) in Mind and Anatomy 4D enable stu- 
dents to explore their brains and body organs; EON Experience offers content for 
learning history and physics (Lynch, 2017). VR gives the possibility to “visualise 
and simulate events that are not perceivable in real life” (Ott & Freina, 2015, p. 6). 

In each of these scenarios, learners at all levels use VR applications created for 
them as a tool for furthering their own subject matter understanding. The TalkTech 
2017 project examines the process of creating one’s own VR content to develop 
information technology literacy skills and explore applications of VR in business as 
future entrepreneurs. 


3 Designing VR with the CoSpaces Platform 


3.1 CoSpaces in Education 


Web-based VR authoring tools such as InstaVR, WondaVR, and CoSpaces make it 
possible to create original VR artefacts with little or no programming experience. 
This study chose CoSpaces because of its simple visual, drag-and-drop interface 
and built-in support for use in educational settings. “CoSpaces Edu offers a com- 
plement to traditional teaching methods by immersing students into a world where 
they can create anything in 3D, learn coding while having fun, and connect with the 
curriculum on a completely new level” (Delightex GmbH, 2017). 

Virtual reality systems promote situated learning through the immersive expe- 
rience of interactive objects, environments and processes (Greenwald et al., 2017). 
CoSpaces “allows you to create virtual 3D worlds that can be explored using 
smartphones, tablets, and PCs with the ability to take advantage of the VR viewer” 
(Bertolini et al., 2018, p. 121). CoSpaces uses a visual programming editor similar 
to the Scratch programming environment, to specify code for modelling and ani- 
mating simple virtual worlds. In primary and secondary education, students have 
used CoSpaces for digital storytelling, creating virtual art exhibitions (Bertolini 
et al., 2018) and recreating historical scenes (Boedo Naya, 2017; Krause, 2017). 
Creating VR lessons accelerates learning by allowing students to apply their own 
subject-matter knowledge. 

CoSpaces allows users to select from provided scenes or upload original 
360-degree or panoramic images to use as backgrounds for their VR artefacts. They 
enhance the scene with sprites, objects, gestures, and interactions. After designing, 
developing, and coding their virtual worlds with the CoSpaces web app, students 
can explore their virtual worlds using mobile devices with the CoSpaces mobile 
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app, available for Android and iOS devices. The mobile app enables users to visit 
their virtual worlds using a Google Cardboard headset or similar VR viewers, 
creating fully immersive experiences. Users may share their worlds with others by 
providing a link, or a QR code, or embedding it on a blog or website. 


3.2. CoSpaces in the TalkTech 2017 Project 


As a requirement of the TalkTech 2017 project, students at each university 
researched the same business or industry in their country as their international 
partners and created a related VR scene to share. Students then viewed their 
partners’ VR scenes and compared them with their own experience of similar 
scenes in their home country, providing a glimpse into another country’s culture. 

For example, the team learning about VR applications in the restaurant industry 
each prepared a VR scene at their local Starbucks coffee shops, as shown in Fig. 1. 
They captured the background image of the facility using a 360-degree camera app 
on a smartphone, and then imported the image into CoSpaces to add additional 
content and context. In both cases, animated sprites greet the customer. The 
Romanian students added engaging conversation in chat bubbles while spinning 
yellow stars highlight the bakery items or coffee selections. The American team 
added a unicorn and a spinning jewel to their scene. Figure 1b shows how a VR 
artefact created in CoSpaces appears in a browser, and on a mobile device to be 
inserted into a Google Cardboard VR viewer. 


Fig. 1 Starbucks coffee shop a in Romania, b in Google Cardboard, and ¢ in the United States, 
created in CoSpaces VR 
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Fig. 2. Creating CoSpaces VR for tourism a in Romania and b in Boston 
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Fig. 3. Blockly code for animating scenes in CoSpaces 


The experience of Fig. la and b is available at https://goo.gl/Uc4ETT using 
CoSpaces, or with a short video at https://flipgrid.com/be7f23. 

In another VR scene created by students investigating applications of VR and 
tourism, students created VR experiences for local landmarks. Figure 2 shows a VR 
scene at a public square in Romania and the United States. 

Figure 3 shows a portion of Blockly code to animate some sprites in Fig. 2a. 
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Fig. 4 CoSpaces tours of campus libraries a in Romania and b the United States 


Figure 4 shows scenes that each group created for promoting campus recruit- 
ment by offering virtual tours of the libraries at their respective schools. Both scenes 
employ sprites that welcome the visitor and provide information about the library 
and its facilities. 


4 TalkTech 2017 Implementation 


A cohort of 67 students (37 honours students in a first-year introductory 
Information Technology course at Bentley University in the United States, and 30 
students enrolled in a Technologies of Multimedia course in their fourth year at 
Politehnica University of Timisoara, Romania) participated in this study. Of those 
who participated, 43 students (completed a survey about their experiences at the 
end of the project. All students had some previous experience using the web, 
collaboration tools, and mobile devices. Participants formed self-selected groups of 
four of five (two or three students from each country) collaborate with their 
international partners. Students were mostly between 21 and 23 years old (46%), 
and between 18 and 20 years old (54%). The Romanian students were, on average, 
about 3 years older than their American partners. The authors recognize that 
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differences exist in age, programs of study, technical abilities, and class sizes of 
students at each university. For the purposes of this research, the authors embraced 
these differences to design a learning project that encouraged collaboration among 
group participants. The project’s common language is English. 

This study implements a project-based learning approach to introduce VR 
concepts by engaging students to create original VR content related to a business or 
industry. In a project-based learning activity, “learners construct knowledge by 
solving complex problems in situations in which they use cognitive tools, multiple 
sources of information, and other individuals as resources” (Blumenfeld et al., 
1991, p. 371). 


4.1 Exploring Uses of VR in a Business 
and Technology Context 


The TalkTech 2017 project engages students to select and research applications of 
VR in one of these businesses or industries: culture and tourism, technology retail, 
fast food/restaurants, sports, university recruitment, and then create their own VR 
scenes to share with their partners. The learning scenario enables students to 
demonstrate several technology literacy skills and develop subject-related knowl- 
edge as they create and share their original VR scenes. 


e Research how an assigned industry or enterprise to see examples of how it might 
use VR 
Create a background image for the VR scene 
Design and implement avatars, sprites, or other virtual content and their inter- 
actions with this VR scene using CoSpaces 
Test on a mobile device, or with Google Cardboard headset 
Create a demonstration video on FlipGrid, a video sharing platform, and 
instructions for sharing with international partners 

e Share the VR artefact and related research on ZeeMaps, a collaborative mapping 
tool 


Completing these steps requires using search engines, transferring files from one 
device to another, interacting with content in multiple file formats, navigating the 
Web, and learning basic coding concepts such as performing steps “in parallel’, 
looping, actions, and interacting with objects. Students must choose which tools 
they will use to interact with their partners and manage the project, from syn- 
chronous communication by voice and video conferencing to asynchronous chats 
and email; file sharing, multimedia creation, and project management. 
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5 Findings 


This section summarizes student prior experience with and attitudes toward learning 
about VR, the importance of VR in business, and a desirability test reflecting 
student satisfaction with the TechTalk project. 


5.1 Experience and Attitudes Toward Learning About VR 


When asked which tools they used for VR, all students identified CoSpaces, and 
many of the American students also said they had experience using Oculus Rift 
because a related assignment required them to experiment with that headset. Prior to 
this project, fewer than half of the students claimed to have personal experience 
using AR or VR tools, as shown in Fig. 5. 

A survey question after the project ended asked students about their perception 
of the usefulness of AR and VR technologies. Results, shown in Fig. 6 suggest that 
the majority found AR and VR to be easy to use, clear and understandable, pro- 
ductive, and educational. 


Prior to this project, to what extent have you 
used AR/VR? 


12 

8 
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Fig. 5 Prior experience with AR/VR 
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Fig. 6 Attitudes toward AR/VR 
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Attitudes toward ICT tools with respect to Business Success 


It is possible to work effectively over the Internet 
with people I never met in person 


This project helped me experience what is involved 
in being part of an international business 


This project helped me learn what is involved 
in starting an ICT business 


It is important understand web-based collaboration 
tools in order to succeed in business 
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Fig. 7 Attitudes of TalkTech 2017 on using ICTs for business success 


Students commented about applications of VR in the industries they studied, and 
the promise that VR holds. Said one student, “Virtual reality ... [gives] its users a 
simulated experience of what it would be like to be in certain spaces or environ- 
ments. For example, I may use virtual reality technology to go on a trip to Venice, 
Italy without ever setting foot outside my home.” They recognize the need for 
collaboration and being productive in virtual environments: “Above all, the project 
provided me with an experience which gave me a better understanding of collab- 
orative work in the business world. It is not always that business people will be in 
the same city, state, or country. Therefore, as difficult as it may be, it becomes 
necessary for individuals to find ways to maintain productivity and efficiency within 
their fields of work.” 

Figure 7 shows students’ attitudes on using technology and collaboration tools, 
being members of international teams, and how useful they think these experiences 
will be in contributing toward their future careers. Most students agree or strongly 
agree that participating in the TalkTech project taught them about culture, collab- 
oration, working on international teams, and the importance of web-based collab- 
oration tools in an information/technology business. 


5.2 Desirability Test 


To measure how desirable students found this experience, the authors followed a 
selection exercise developed by Benedek and Miner (2003) based on a set of 40 
words, both positive and negative, covering a variety of dimensions. To account for 
any bias to give positive feedback, at least 40% of the words were negative. After 
the project, students selected five words that best describe their experience 
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Fig. 8 TalkTech 2017 desirability word cloud 


participating in this project. Students ranked the words they selected on a scale from 
1 to 5, where | is the most precise. This method presents results visually in the word 
cloud in Fig. 8. 

Students chose the words that appear in a larger font more frequently than words 
shown in smaller fonts. The four words in grey in the largest font size (stressful, 
frustrating, time-consuming, rigid) are the most often selected words that describe 
the students’ experience in this project in a negative context. They found it 
“stressful” and “frustrating” at times dealing with differences in time zones, slow 
Internet connections, and occasionally non-responsive international partners; 
“time-consuming” reflects that several students felt that completing the project took 
longer than they had anticipated; “rigid” suggests that some students felt the 
requirements were very strict despite their open-ended nature. 

The most popular positive words were collaborative, fun, useful, and attractive. 
Collaborative describes their experiences with many of collaboration tools used and 
the opportunity to be a member of an international team. “Fun” describes students’ 
engagement with the project: they had the opportunity to get off campus to visit a 
local business or place of interest, and view completed VR scenes in Google 
Cardboard. Useful hints that the students recognize the value in becoming familiar 
with VR technologies and their applications for their own future careers. Attractive 
suggests that the project was at the appropriate level, and that students learned new 
skills by participating. 
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FUTURE USES OF VR 
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Fig. 9 Future uses of AR/VR 


5.3 Future Uses of AR/VR: From an Entrepreneurship 
Point of View 


When asked for what purposes they might use AR/VR apps in the future, students 
recognized the large market of VR in the gaming industry. Others noted retail and 
educational applications, and some simply would use VR “because it is cool” as 
shown in Fig. 9. 

Students recognized the potential of VR in future business opportunities espe- 
cially in the areas of advertising and marketing. Comments included, “we can 
advertise with this”; “I learned how to include information in these experiences, 
which would be helpful to advertise a product”, “AR/VR are tools that can be used 
to have products and ideas stand out. Therefore, AR/VR can be used as marketing 
tools.” and “[I learned] different tools for marketing, partnering with strangers and 
working together on a project.” Students came away with practical possible future 
applications, such as “a restaurant which you can visit in Virtual Reality and have 
the menu appear on your phone screen when you scan a QR code.” Another student 
said, “VR can be implemented by a business to immerse a customer in any 
experience or environment that the business chooses.” 

When asked, after completing this project, “would you consider using AR/VR if 
starting a new business or start-up?” students varied in their enthusiasm for these 
technologies. Responses included “VR and AR can reduce language barriers [be- 
cause of their visual nature]”; “they make it easier to give customers information”. 
One student said he would use AR or VR “only as long as it would enhance the 
product or experience for the customers.” Another student was sceptical of the 
technology in its current state, saying “There is definitely potential (especially in 
AR), but right now the products don’t seem very efficient for whatever practicality 
they provide, so I would not use AR or VR.” 


6 Discussion and Conclusion 


The process of creating original VR scenes was new for most students and provided 
a useful way to apply knowledge gained by researching VR applications in busi- 
ness. Said one student, “Virtual reality is often linked to applications that deliver 
virtual immersion in high-definition 3D visual environments. Such applications 
include, for example, CAD software, display acceleration hardware, head mounted 
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displays, electronic gloves for databases, electronic clothing, and more.” Students 
recognized the potential of this technology in industries beyond gaming, the one 
with which they were most familiar. 

Creating VR also offers a new way for students to use their mobile devices. 
Researching panoramic camera apps, using a 360-camera connected to a smart- 
phone over Bluetooth, designing or running the CoSpaces app with Google 
Cardboard, all incorporated the use of mobile devices to this project. Students found 
CoSpaces to be “very user-friendly” and most students, even with little or no coding 
experience, were able to add basic animations to their scenes through the visual 
Blockly development tools. Although challenged by using new apps and tools, and 
not always having the tech skills needed to apply them, all students prevailed in this 
project, and often relied on their team members, the Internet, and their instructors 
for additional information and guidance when necessary. 

The TalkTech 2017 project introduced VR as a relevant current technology, and 
as the goal for student-created multimedia. The authors conclude that TalkTech 
2017 provides a project-based learning environment in which students develop and 
demonstrate their technology literacy skills, while learning about VR and its 
business applications. Students learned that business applications of VR will change 
how companies present and sell products and how customers experience them, 
across several industries. Students also learned that creating, sharing, and exploring 
VR scenes can change how they experience virtual worlds, as well as how they 
apply their knowledge gained in those virtual worlds, to the real world. 
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Immersive Virtual Reality Mm) 
(IVR) in Higher Education: oe 
Development and Implementation 


Paula Hodgson, Vivian W. Y. Lee, Johnson C. S. Chan, Agnes Fong, 
Cindi S. Y. Tang, Leo Chan and Cathy Wong 


Abstract University educators anticipate better engagement with students through 
the adoption of the three-dimensional visualization made possible by immersive 
virtual reality (IVR). Materials can be captured in 360° video for viewing through 
smartphones bracketed in head-mounted displays (HMDs) with motion sensors. 
Alternatively, materials can be viewed on notebook computers and tablets to offer 
some degree of VR experience. The paper reports on the first two undergraduate 
courses that have adopted both VR and IVR modes for classroom learning: 
‘Pharmacology and Therapeutics’ and ‘Understanding Ecotourism’. The 360° 
videos have undergone a complete cycle of design, development, implementation 
and evaluation. These video captures can transcend physical boundaries in both 
clinical cases simulating a hospital ward and natural countryside landmarks. 
With VR and IVR embedded in the classroom, students expressed greater learning 
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satisfaction while experiencing more opportunities to rehearse professional skills 
and explore historical artefacts with deeper cultural understanding. 


Keywords Immersive virtual reality - Higher education - ADDIE 


1 Introduction 


Virtual reality (VR) is defined as ‘a real or simulated environment in which a 
perceiver experiences telepresence’ (Steue, 1992: 76). Users may visualize from 
two to three dimensions of an environment or an object, experience simulated 
situations to explore virtual locations as preparation for or replacement for actual 
visits or field exploration, interact with simulated clients or patients, simulate 
experiments and processes, and take actions while interacting with the virtual 
environment or object. VR can be in the form of a non-immersive mode, presented 
in a screen setting showing programmed scenarios for stakeholders to rehearse 
interactions in safe and naturalistic environments (Kandalaft, Didehbani, Krawczyk, 
Allen, & Chapman, 2013), and there is the possibility for users to explore desti- 
nations in the context of virtual tourism (Huang, K. F. Backman, S. J. Backman, & 
Chang, 2016). Putting virtual 3D images in a screen setting (whether it is a laptop or 
desktop) means that users can access without additional tools. 

However, immersive virtual reality ((1VR) enables users to have an experience 
that perceptually surrounds them and gives them a sense of presence or actually 
being within it. This can be of two types: a computer automatic virtual environment 
(CAVE); and head-mounted displays (HMDs), sometimes equipped with motion 
sensors. The first type is set in a room in which computer-generated graphics are 
projected onto the walls and users’ head and hands are tracked as they interact with 
the simulated environment in the VR room (Cruz-Neira, Sandin, DeFanti, Kenyon, 
& Hart, 1992). There is a second type of virtual experience, without being confined 
to a space with projectors, in which learners can interact with the immersive 
learning environment by wearing an HMD. 

IVR may also offer simulated settings to provide a virtual tour with panoramic 
views of destinations; allow students to practise the procedural steps in a physical 
environment, such as operating an overhead crane (Huang, 2003); function in daily 
life activities at home (Lee et al., 2003) or away from the home environment 
(Grewe et al., 2013); learn physical movement (Bailenson et al., 2008), and train 
how to deal with hazards such as radiation (Koepnick et al., 2010). Research has 
shown many positive outcomes through applying VR for rehabilitation purposes 
(Grealy, Johnson, & Rushton, 1999; Grewe et al., 2013; Lee et al., 2003). 
Applications of IVR have been used in museum exhibitions to illustrate archaeo- 
logical artefacts (Bruno et al., 2010); in flight simulation with low latency and at a 
low cost (Yavrucuk, Kubali, & Tarimci, 2011); and the adoption of VR storytelling 
techniques in journalism (Hodgson, 2017). More importantly, there is growing 
interest in adopting VR or IVR in university classrooms. Fagan, Kilmon and 
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Pandey (2012) find positive acceptance of VR simulation in nursing education. This 
type of learning experience can be applied in many disciplines, including life 
sciences, education, law, architecture, archaeology, history, geography resources 
and management, nursing, and business. 


2 Adoption of IVR in Higher Education 


Both three-dimensional and 360° VR video capture have created an alternative mode 
of learning experience in universities. A comprehensive virtual tour can be constructed 
in a non-immersive mode, including images, videos and interviews with key stake- 
holders when students need to explore core concepts and practise skills in a course 
(Patiar, Ma, Kensbock, & Cox, 2017). In addition, 360° viewing may provide a full 
panoramic view for students to be engaged in a novel environment in which the 
location may be remote, difficult to reach, or politically unstable (e.g. archaeological 
sites). Students can rehearse procedural steps or examine virtual sites close up, with a 
bird’s-eye view as they build new concepts in disciplines. Moreover, students can 
‘visit’ locations and observe natural phenomena across seasons and in different 
weather conditions (e.g. field trips exploring nature reserves). Chang, Lin, and Hsiao 
(2009) find that earth science students appreciate the VR option as preparation for 
actual field visits; students can use a 3D building tool to undertake exploration and 
apply theories to Google Earth for geography education (Krakow, 2012). Tawhai 
(2017) advocates that IVR can be used for teaching of forensic investigation. Students 
may develop transferable professional skills through revisiting crime scenes to make a 
variety of attempts when learning how to carry out a crime investigation. 

IVR settings also allow visualizing and testing of abstract concepts, such as 
manipulating cells from different structures at the microscopic level, simulating 
muscle responses during dissection while exploring abstract concepts, or con- 
ducting virtual laboratory experiments when there are ethical concerns. IVR can 
also be applied in laboratory settings, and students can rehearse through rich-media 
presentations in VR before working on high-end laboratory equipment, which may 
not be accessible at all times. By maximizing the learning opportunities, engi- 
neering students have a sense of the product through 3D visualization before 
producing an actual prototype (Abulrub, Attridge, & Williams, 2011). 

University students are expected to have fluent communication skills, and Harris, 
Kemmerling and North (2002) report that they build confidence in public speaking 
as one of the generic skills if these can be practised first in a virtual setting. 
Moreover, students attending professional programs are required to acquire pro- 
fessional manners of communication with stakeholders. For instance, medical 
students are required to communicate well with patients having different kinds of 
medical condition, where responses are often affected by a state of emotional 
distress. Through the virtual environment, they can practise and rehearse commu- 
nication skills with a virtual patient, family members and other professional prac- 
titioners before dealing with real people. Beggan, Morton, and Simpson (2017) find 


164 P. Hodgson et al. 


that nursing students may build empathy when they experience being patients in a 
busy ward setting through 360° video VR clinical scenarios. A head-tracking sensor 
built into an HMD allows users to move their head to interact with full-vision 
immersion. 


3 VR/IVR Design and Development: 
A Teaching-Development Grant Project 


In Hong Kong, university educators have started to venture into the emerging 
immersive learning environment. Adopting the Assessment, Design, Development, 
Implementation and Evaluation ADDIE model, this paper reports a university-wide 
project that is supported through the Teaching Development and Language 
Enhancement Grant funded by a university in Hong Kong. The ADDIE model is 
still one of the most widely adopted for designing and developing prototypes for the 
virtual learning environment (Soto, 2013). This project aims to support academics 
on using VR/IVR to provide an enhanced learning experiences in disciplines. 
Design and development of each VR/TVR will include an initial phase of proto- 
typing, followed by fine-tuning after a small-scale pilot with students, followed by 
implementation in class and evaluation of the learning experience immediately after 
the class. As a university-wide project, we aim to support six courses in depart- 
ments through a funding of HK$800,000 (circa £72,700) between December 2016 
and July 2018. Two courses are reported in this paper because they are the first two 
sub-projects to have completed the full cycle. The remaining four sub-projects are 
still in the development phase. This paper will examine the two courses that have 
embedded IVR in class to determine whether in-depth inquiries are being triggered 
through the novel learning setting. There is variation in the production process of 
the two sub-project types of VR/IVR that have completed the full ADDIE cycle, 
while ‘Pharmacology and Therapeutics’ is set for an indoor scenario and 
“Understanding Ecotourism’ is set for an outdoor scenario. 


3.1 Designing and Developing VR/IVR Cases for Practising 
Professional Clinical Skills in ‘Pharmacology 
and Therapeutics’ 


Pharmacy students in Hong Kong face a common challenge of having limited 
opportunities to experience professional clinical practice, even though they are 
required to interpret clinical cases and attend pharmacy ward rounds during Years 3 
and 4 of their study. There is a chronic lack of systematic teaching materials for 
pharmacy students on the preparation of clinical cases, the interpretation of clinical 
notes and clinical abbreviations, and the assessment of clinical cases. 
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Aligned with the teaching content in the program, the VR/IVR cases were 
constructed to prepare Year 3 pharmacy students before they attended clinical 
settings the following year. Staff from the Pharmacy Department produced a sto- 
ryboard of two scenarios with patient backgrounds extracted from authentic cases in 
Prince of Wales Hospital. The scenarios simulated the process of consultation with 
a patient and with a medical team of doctors. Screenshots were set and agreed with 
the staff on the storyboard with a view to being a pharmacist interacting with the 
professional staff and the patient. The scripts were sent to ‘actors’, who were invited 
as doctors and the patient before the date for filming. The technical team took shots 
of a clinical scene setting on campus. Six cameras held in a rig were used to capture 
a full spherical indoor view: nine shots to capture case | and seven shots for case 2. 
The best captures were selected and converted for the IVR cases by the technical 
team, with occasional review meetings by both teams. A pilot was conducted with a 
few pharmacy students to collect feedback on their experiences when viewing with 
smartphones connected to VR headsets and tablets in a web-based environment to 
test the feasibility of the whole setup for teaching (Fig. 1). To provide a better 
understanding of this type of learning experience, the teacher was also invited in the 
pilot so that she could have a personal experience with the HMD with sensors 
because she planned to introduce a totally new method in her teaching. During the 
class, one student walked through the scenario with the HMD and sensors, while 
the professor explained the clinical diagnostic processes (Fig. 2). 


Fig. 1 Conducting a pilot with some pharmacy students on 360° viewing mounted in Google 
cardboard boxes 
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Fig. 2. Student using HMD 
with sensors while the 
professor explained the 
clinical diagnostic processes 
to the class 


3.2 Designing and Developing VR/IVR Outdoor Field Trips 
for Students Taking ‘Understanding Ecotourism’ 


A two-credit course on ‘Understanding Ecotourism’ has a field trip as part of the 
learning activities in addition to lectures and invited guest talks on campus. 
However, only one field trip is feasible per semester. Therefore, virtual field trips 
can provide a wider exposure to other ecotourism sites while students gain extended 
knowledge in the course. As a pilot, only one virtual field trip was set up, to Lai Chi 
Wo, a popular attraction with historical heritage and declared monuments. An initial 
meeting was arranged to discuss the route and foci of the virtual field trip with the 
teaching faculty, while samples of 360° video capture were shown. Site shooting by 
the filming crew and the teaching faculty was arranged. Much work on film editing 
was done after the trip, and hot spots with additional images and historical infor- 
mation provided by the teacher were added to the 360° video-based educational 
site. Markers were created so that students could navigate from one location to the 
next. A pilot test was set up with a few students in the same teaching classroom to 
collect feedback on viewing with different mobile devices and their comments on 
improvements to the prototype. HMDs were not intended to be used with over 50 
students in the class, and students were given full instruction before the class to 
bring their notebook computers or pre-load the videos to their smartphones for 
viewing with cardboard boxes. The notebook used by the teacher was also 
pre-loaded with the VR video capture before class. The professor first made initial 
guided discovery in the lesson, illustrating the virtual tour with the screen projector 
(Fig. 3). However, students could view the virtual site in their own time at https:// 
s3.amazonaws.com/aws-website-laichiwo-trial-7mewp/index.html. 
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Fig. 3. Showing the virtual tour to Lai Chi Wo in class (left-hand panel: satellite view; the 
high-resolution 360° images are displayed in the right-hand main frame) 


4 VR/IVR Implementation in the Classroom 


Huge bandwidth is required to transmit high-quality VR videos (He, Hu, Jiang, & 
Li, 2018). The learning experience will be hampered if students cannot view the VR 
videos during class time. Therefore, to lower the demand on Wi-Fi access to the VR 
materials through the smartphones that students normally bring to class, they were 
given full instructions to download the video and application to view VR materials. 
In addition, six tablets preloaded with the VR teaching materials and thirty sets of 
VR headsets were delivered to the venue for the teaching sessions. An hour before 
the teaching session, a high-speed notebook with the HMD and sensors was pre- 
pared in the locations for the teaching sessions on pharmacy. 


4.1 Pharmacy Class 


Two VR clinical cases were developed on the topic of acute coronary syndrome and 
heart failure for the class with 38 pharmacy students. An hour-long lecture was 
conducted before the one-hour VR/IVR teaching session for students to learn 
conceptual understanding of pharmacological treatment with patients having these 
types of medical complication in a VR clinical context. As it was the first time for 
the teacher to adopt the VR technology in class, one student put on the HMD and 
sensors to go through the IVR case with the video displayed in a front screen for the 
class, and the teacher explained the medical concepts and paused between questions 
for students to think and reply. Tablets and VR headsets were set aside for students 
to use. Student feedback was collected after the first class of VR implementation. 
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Materials and case questions on the second topic were sent to students before class 
so that they could review and prepare for the VR case discussion. Students were 
divided into small groups (four—five students per group) to facilitate in-class 
discussion. 


4.2 Ecotourism Class 


One VR video-capture field trip to Lai Chi Wo in the north-eastern New Territories 
of Hong Kong was produced based on the real environment in June 2017. The VR 
materials were used in a university general education course with over fifty stu- 
dents. The two-hour session was split, with the first hour conducted by a guest 
speaker from the Agriculture, Fisheries and Conservation Department presenting a 
talk on ecotourism. The VR session took place afterwards. The teacher showed the 
VR with a front screen display, and his students could view the site either with their 
own notebook computers or tablets, or using VR headsets with their smartphones. 
The teacher first walked through some checkpoints to illustrate ways of viewing 
while students watched individually. They were then given time to work in small 
groups to go through the rest of the virtual tour with digital devices to discuss 
questions and issues relating to ecotourism, such that they had to think critically 
about different perspectives while viewing the high-resolution 360° video capture 
with embedded information. The VR platform was set to provide an alternative 
opportunity for students to experience and study the environment in Lai Chi Wo 
without an actual site visit. This served to prepare students with skills on obser- 
vation and critical awareness when they went on a field trip to a different location. 


5 Evaluation of the IVR Project 


Various stages of evaluation are crucial to informing an innovative dimension 
embedded in classroom teaching. Evaluation becomes a focal point, from the design 
and development stage to classroom implementation. Feedback by teaching faculty 
on the storyboard was sought and confirmed before shooting was arranged. The 
technical team, including an instructional designer and two technical staff, held 
reviews periodically of prototypes of sub-projects, ‘Pharmacy’ and ‘Ecotourism’, 
during the VR production stage. Students were involved in key stages, from testing 
to post-class evaluation. In the testing stage, their comments guided the technical 
team in choosing types of VR headset from Google cardboard boxes to other 
handheld VR headsets. Their suggestions provided fine-tuning of the VR devel- 
opment before the VR sessions were launched. 

Student interviews were conducted after their first VR experience in the 
Pharmacy class. Many remarked that they were overwhelmed by all the information 
provided in the first VR case and felt some degree of disorientation, not knowing 
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how to respond to some of the questions during case discussion. Having reviewed 
their comments, this brought forward re-examination of the process of in-class 
teaching. Subsequently, case questions and materials relating to the second topic for 
the second VR case were sent to students to study in advance before attending the 
class. A survey of students was conducted after they had experienced two VR 
sessions, and 34 responses were collected. 

A seven-point Likert scale, with 7 ‘strongly agree’ and | ‘strongly disagree’, was 
used in the item statements and students had very positive responses (Table 1). 

Students commented that ‘good with images of patient’s clinical symptoms’, 
‘the VR allows us to see the patient non-pharm aspect’. However, some commented 
that ‘the technical problems may cause waste of time’, and ‘can skip the part with 
VR glasses’. One student suggested that ‘It would be nice if there are notes which 
summarize what to monitor for each of these diseases and the ideal dosage of the 
drugs next to the case studies’. 

While only one VR session was held for the Ecotourism class, a post-class 
survey with a seven-point Likert scale was also conducted on their perceptions of 
the VR experience on the virtual tour, but only 17 responses were collected. Among 
them, 88.2% had not visited Lai Chi Wo. Feedback on these students is shown in 
Table 2. 

Here are additional comments expressed by few students: ‘feeling very dizzy’, 
‘to include audio explanation in various checkpoints’ and ‘good experience, and 
appreciate the technical team support’. 


Table 1 Pharmacy students’ feedback on experiencing the VR sessions 


Survey items Strongly | Agree | Fairly | Neutral | Fairly Disagree | Strongly 
agree (%) agree (%) disagree | (%) disagree 
(%) (%) (%) (%) 


I am alert to 


non-pharmaceutical 
management (e.g. patient 


responses) in the VR clinic 


41 


29 


27 


Discussion of VR cases can 


enhance my learning 
interest 


65 


I feel that I am better 
prepared for the clinical 
practice in the final year 
with the VR clinic 
experience 


50 


27 
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Table 2 Students’ feedback on virtual visit to Lai Chi Wo 


Survey items Strongly Neutral | Fairly Disagree | Strongly 
agree (%) disagree | (%) disagree 
(%) (%) (%) 


The VR trail 41 
enables me to 
visualize the site 
I enjoy the IVR 15 
headset to view 
the trail 

I enjoy the tablet/ | 9 


web nIVR to view 
the trail 


I have interest to | 18 
find out more 
about the VR trail 
site after class 


I consider visiting | 24 
Lai Chi Wo in my 
own time 

I appreciate the 35 41 18 0 0 0 6 
VR experience as 
part of my 
university study 


6 Discussion 


Wi-Fi access has been made available in university campuses in Hong Kong, so the 
majority of students can view materials on learning management systems using 
their mobile phones or notebook computers for academic study. However, leaping 
to viewing high-quality video with VR headsets for immersive experience or tablets 
and notebook computers for in-class activities may require much preparation. 
Students need to be familiar with the new tools and require more time in preparation 
before starting the first class. Although students can be provided with full 
instructions to download viewing applications and VR video to their mobile phones 
before class, VR headsets with sensors are not readily available in classrooms. If 
students prefer to watch with their notebook computers, bandwidth in a classroom 
may be limited for transmitting high-quality videos. Choices of alternative viewing 
modes need to be considered when staff use VR/TVR teaching in a classroom. 
Nevertheless, a VR environment goes beyond the virtual presence and there is an 
affective dimension in which emotions are induced when learners interact through 
the medium (Riva et al., 2007; Huang, Cerekovic, Pandzic, Nakano, & Nishida, 
2009). Bringing a clinical ward setting into a traditional classroom, students expe- 
rienced first-hand clinical exposure in class with guided, step-by-step VR teaching 
materials on clinical cardiology pharmacy while translating clinical knowledge into 
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practice. Patient-centred treatment can start in class, because they can learn to 
observe symptoms presenting in a patient and practise communication with virtual 
doctors. Although students can visualize patients in a ward, they start to broaden 
their scope of learning pharmaceutical treatments from reading paper-based cases 
with information on medical history to focusing on patient’s observable physical 
conditions with reference to medical data collected. With pre-clinical preparation, 
they may draw a closer connection between academic study and the complexity in a 
hospital ward that they face during the internship in the final year of the pharmacy 
curriculum. Educators may need to reconsider ways to maximize the new learning 
environment when adopting IVR in a classroom (Iglesias & Galvez, 2008). As 
students can view the IVR videos with VR headsets individually and can form 
groups to work on VR scenarios, educators can consider converting the lecture mode 
with VR video projecting on a front screen to interactive group tasks providing that 
personal immersive experience. 

Moreover, research findings support positive learning outcomes: learners can 
find it easier to understand abstract concepts in three dimensions, bridge space over 
time, and examine trends and changes (Fernandez, 2017). And 360° video capture 
on a virtual tour can provide students with a space to explore and examine the 
virtual field when it is not readily accessible, or have preparations to review 
materials in a virtual setting before conducting a thorough study on site. However, it 
is resource-intensive to create a single 360° VR video, although educators can make 
use of publicly accessible, high-quality, 360° video galleries in 360 cities (https:// 
www.360cities.net/video_gallery). In addition, high-quality, 360° VR videos are 
produced by journalists as immersive news (http://bbcnewslabs.co.uk/projects/360- 
video-and-vr/). Depending on the intended learning outcomes and how educators 
design the VR content, students may observe passively or explore actively when 
they discuss and make critiques between peers in the VR environment (Black, 
2017). 


7 Conclusion and Recommendations 


This is the first institution-wide project creating VR/TVR educational materials for 
classroom teaching. The project has been centrally funded, and therefore knowl- 
edge in producing the VR/IVR projects can be channelled across departments. 
Second, adoption of emerging technologies in teaching can encourage better stu- 
dent engagement, as evidenced in various stages of student evaluation, although 
there is much higher demand in time for early adopters of VR and IVR technologies 
in the process of planning, design and development for classroom implementation 
(Iglesias & Galvez, 2008). Although HMDs with sensors are becoming more 
affordable, it is still an add-on device to be arranged before class. With advances in 
integration of artificial intelligence and new models of VR devices on the market 
each year, the mode of interaction with VR materials is still evolving. As reported 
in this paper, interactions through VR materials can serve to build professional 
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competence and instil interest and curiosity while exploring various virtual tours, 
whether 360° videos are adopted in VR or IVR mode. To provide a sustainable 
impact on student learning, educators can set a variety of learning tasks with the VR 
materials from different perspectives and foci before and after in-class 
implementation. 

As tools for building VR materials are emerging in the market, learners are no 
longer just consumers. Instead, they can take part in the creation of VR materials so 
that more creative and advanced VR objects can be built to gear up the next 
generation of mixed reality native. 
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Cultural Heritage Objects in Education Mm) 
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Abstract In this paper, we focus on the use of cultural heritage objects (historical, 
natural and technical) in the education process. We create universal content that can 
be used in education in schools, but also at home with the use of virtual and 
augmented reality. We discuss the potential of the VR and AR tools for better 
memorising of the information based on the virtual objects and scenes. We describe 
the selection of individual objects in order to be able to communicate information 
about individual objects in pairs across the Slovak-Ukrainian border and to use 
them appropriately on the tools created in the InovEduc project to improve the 
education at secondary schools on subjects like history, geography, religion, 
regional education, civics and languages. We also focus on creating content so that 
it can be used in other areas such as tourism, museums and other cultural 
institutions. 


Keywords Education - Cultural heritage - Virtual reality - Augmented reality 


1 Introduction 


The use of modern digital technology in the teaching process is currently a modern 
approach to education at various levels of schools. In terms of acquiring knowledge, 
the memorability of the information obtained is also an important factor. Virtual 
and augmented reality technologies can be increasingly integrated into the learning 
process and profited by an innovative approach to learning. In this paper we present 
the possibilities of using digitised objects of cultural heritage mediated through 
virtual and augmented reality in the teaching process. The use of virtual and aug- 
mented reality in teaching has begun to be considered from the very first prototypes 
of the device. Virtual reality is used to create different interactive learning envi- 
ronments in different areas. As Thorsteinsson (2013) teaches using virtual reality, 
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it is possible to create a virtual reality learning environment that promotes online 
communication between students and teachers. A similar solution is also being 
promoted by Monahan, McArdle and Bertolotto (2008). It is also possible to use 
virtual reality for teaching within specific domains, etc. as stated in Machado, 
Moraes, Nunes and Costa (2011) for teaching medicine, or Hafner, Hafner, and 
Ovtcharova (2013) in engineering education. Education based on virtual reality is 
also possible for disadvantaged groups, as stated in their article Ip et al. (2016). As 
stated in Lee (2012) augmented reality is one technology that dramatically shifts the 
location and timing of education and training. The experimental results show that 
augmented reality based mobile learning system is able to improve the students 
learning achievements as stated in Chiang, Yang and Hwang (2014). Cultural 
heritage objects can be used for teaching through modern ICT. As stated in Ott and 
Pozzi (2011), ICT can really provide any added value to Cultural Heritage peda- 
gogy, education and learning. As stated in Angelopoulou et al. (2011), one of the 
most promising aspects of augmented reality is that it can be used for visual and 
highly interactive forms of game-based learning. Devices that are used for aug- 
mented reality education can be simple and easy to use (smartphones, tablets) or 
may require special setup as mentioned in Novotny, Lacko, Samuel¢ik (2013). 

Nowadays, students are highly influenced by individual technologies, and 
therefore new forms of education must be sought to respect current trends and 
teaching will be a good experience for students. As stated in Martin-Gutiérrez et al. 
(2015) “Right now, education and teaching institutions try to avoid traditional 
teaching methods despite their validity and successful results, as the interest now 
focuses on more productive methods that may improve the learning experience and 
the students’ intellectual level. Computer technologies have provided a strong 
improvement according to educational tools, allowing development of new teaching 
methodologies.” If they are trained in a traditional way, they are mostly in the 
position of passive recipients of information, with very likely that much of the 
information that comes to them will not be remembered. If, in connection with 
receiving information, students are exposed to an emotional experience, through 
simulation in the virtual world, increases the share of information stored not only in 
the short-term but also in the long-term memory. Results of Pan, Cheok, Yang, Zhu 
and Shi (2006) show that the attitude towards virtual learning is an important 
predictor for learning effectiveness and that the self-efficacy had more influence on 
subject effectiveness while technology efficacy predicts more on general effec- 
tiveness. Thanks to digital technology solutions, it is possible to create positive 
emotions, thanks to feeling embedded into the virtual environment. 

In our work we focus on the use of virtual and augmented reality in the teaching 
process so that the content represented by cultural, natural and technical objects can 
be used for independent learning and teacher-led learning with an emphasis on 
obtaining information and memorizing them through technology-induced emotion 
and we have created a series of applications that are independently deployable in 
teaching, presenting cultural heritage in museums, right on landmarks, or in home 
entertainment that share the same content. As stated in Dunleavy and Dede (2014) 
AR is primarily aligned with situated and constructivist learning theory, as it 


Cultural Heritage Objects in Education ... 177 


positions the learner within a real-world physical and social context while guiding, 
scaffolding and facilitating participatory and metacognitive learning processes such 
as authentic inquiry, active observation, peer coaching, reciprocal teaching and 
legitimate peripheral participation with multiple modes of representation. 

Presentation of digitised cultural heritage objects in virtual and augmented 
reality can have a variety of uses. Most often, we can meet their presentations in the 
area of tourism but also within the museums as presented in Rodriguez, Agus, 
Bettio, Marton and Gobbetti (2015) either in the form of extended information 
about the objects directly on the site where the monument is located, or we present 
them outside the places to attract visitors or provide information about the meaning 
of the monument. In the context of education, it is possible to use cultural heritage 
objects as complementary objects in the teaching of history, religion, geography, 
civic education as well as, for example, languages. The objects themselves are not 
the wearer but the intermediary of important information about historical, technical, 
natural and other contexts. By using individual objects and storytelling, stories 
directly included in the lesson can be conveyed. 


2 Issues 


The one of the goal of learning is to provide relevant information so that it can be 
remembered quickly and efficiently. This kind of surface learning can lead with 
adequate teacher to deep conceptual learning as mentioned in Rillero and Padget 
(2012). An important fact may be whether the information given is related to an 
object, time value, or emotion. We have tried to find a way for students to memorize 
information effectively in connection with a visual experience. As part of the 
“TovEduc” project—Innovative Methods of Education to Promote Partnerships, 
we focused on creating a series of tools that would communicate visual information 
about cultural heritage objects via virtual and augmented reality. The objective of 
the InovEduc project was, through selected cultural, natural and technical objects, 
to point out different and common features that represent the real boundary between 
the two states of Slovakia and Ukraine in the context of their common history, 
gender relations and cultural traditions. The aim of the individual software tools that 
were created within the framework of the project was through their content and 
form to make these questions accessible to the students and to encourage them to 
think critically and to remove the barriers in the thinking of the current young 
population based on the evaluation of individual facts and their memorization, 
which is divided by the border. 

The project was implemented in border areas of Slovakia and Ukraine. Given 
that historically the territory was in the past a common development in terms of 
common statehood, religion, cultural development, it was possible to identify on 
both sides of the border the objects of cultural, natural and technical heritage that 
were digitised. Selected objects, despite their apparent disconnect, point to ethno- 
logical, historical and cultural significance to the common features of the 
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Table 1 List of digitised objects 


Group 


Castle ruins 


Object names 


Zborov Castle, Nevice Castle 


J. Lacko 


Digitisation outcomes 


3D models, ideological 
reconstruction of Zborov 
castle 


Jewish 
monuments 


Jewish cemetery in Topola, Jewish 
cemetery in Uzhhorod, Synagogue in 
Bardejov 


3D models of synagogue, 
gravestones, panoramas 


Greek Catholic 


Church in Topola, Hrabova roztoka, 


3D models, interior 


wooden Smigovec, Inovce and Kostryna panoramas 
Churches 

Orthodox Church in Rusky Potok, Danilovo 3D models, interior 
wooden panoramas 
Churches 


Wine cellars 


Wine cellar in Vel’ka Tifa and Wine cellar 
in Serednie 


3D models, interior 
panoramas 


Objects of folk 


Residential house with water mill from 


3D models, interior 


dwelling Vy8sna Jablonka, The Lemko settlement in panoramas 
Zarychevo 
Castles Chateau in Humenné, Castle in Uzhhorod 3D models, interior 


connected by 
Drughet family 


panoramas 


Technical The Railway viaduct in HanuSovce nad 3D models, interior 
monuments Toplou, Blacksmith in Forge in Lysychovo _ | panoramas 
Important Church in Vel’ka Tina, Horjany rotunda in 3D models, interior 
historical Uzhhorod panoramas 

churches 

Natural European Bison from Poloniny, The Valley | 3D model, panorama 
monuments of daffodils in Khust 


development of the society in Slovakia through Ukrainian interdependence in 
particular periods. As mentioned in Volanska (2017), the objects selected for the 
3D digitization shall demonstrate in what manner the common features of the 
region’s history are more significant than the differences. Within the final phase, 24 
objects (14 from Slovakia, 10 from Ukraine) were selected for the digitisation 
process, which had common elements in the history, the way of using, the religious 
confessions, the type of monument, etc. The selected 24 objects were divided into 
pairs so that it is possible to compare common architectural features and ways of 
using objects on both sides of the common border. Object selection was based on 
consultation with ethnologists and teachers from the regions, as they participated in 
the preparation of worksheets that served as a basis for the teaching process with 
defined application scenarios with virtual and augmented reality. Table 1 shows a 
list of objects divided into common groups by type of use. 

As is evident from Table 1, it was necessary to create an effective way of 
presenting objects so that different types of digitised objects could be displayed in 
virtual and augmented reality. 
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Our goal was to create individual tools with the use of virtual and augmented 
reality so that individual technologies can be directly involved in the learning 
process. Teachers use different approaches, from individual work to collegiate 
student work and various techniques of serving information. We focused on the 
individual work of students and try to solve problems related to how each appli- 
cation should be designed to be used in environment of school, when information 
are interpreted by the teacher as well as independently for students in the home 
environment. 


3 Procedures 


In preparing each application, we assumed that they should have adequate hardware 
at schools. For virtual reality applications, schools should have Oculus 
rift + Oculus touch controllers or HTC VIVE headset. Augmented reality apps 
should be available with Android or iOS smartphones or tablets. However, as it 
turned out, schools are not equipped with sufficient hardware or technical expertise 
to use such applications. For this reason, we decided to create also a web appli- 
cation and a standalone application for the Windows operating system in addition to 
the virtual and augmented reality applications. 

Because we wanted to share the same content through individual apps, factors 
such as the size of individual 3D models and additional objects in the scene, the 
rendering speed of individual devices, and the visual quality of each 3D model were 
needed to digitise cultural heritage objects. As much as possible, we wanted to keep 
the same visual impression within each app. For these reasons, we have decided to 
minimize the use of photogrammetry to create 3D models and replace it where it 
was possible by manual modelling to control the number of individual polygons in 
3D models. Individual models showed the external appearance of individual cul- 
tural heritage objects. For interior design, we decided to take advantage of the 
panorama, because the interior model would require a lot of data for quality display 
in the form of a 3D model. 

Two types of scenes were designed for each application. One type of scene was 
common to a standalone application, a web application, and a virtual reality 
application, and the second type of scene was ready for augmented reality apps. 
Both types of scenes included 3D models of the main objects, interior panoramas, 
or the objects in the form of panoramas for the exterior. The virtual reality scene 
was enriched with terrain, vegetation and a spherical panorama of the sky and a 
cylindrical panorama of the distant surroundings obtained through the drones, as 
you can see in Fig. 1. 

For each type of application, it was necessary to create a custom graphical user 
interface that is based on the need for navigation and object selection in relation to 
the different types of input devices and imaging techniques used. 

Each scene of a particular object is complemented by a number of multimedia 
objects such as descriptive text, a real object image, as well as sound and eventual 
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Fig. 1 Scene of the Chateau in Humenné surrounded by its environment for virtual reality 
application 


Zborov castle 


Fig. 2. 3D model of ideological reconstruction and ruins of Zborov castle 


descriptions directly linked to individual parts of the object in the form of hotspots. 
Most scenes consist of a 3D exterior model and optional interior panoramas. For 
some objects, additional features related to the historical development or func- 
tionality of the object are added. For example, Zborov Castle contains, besides the 
photogrammetrically processed current state of ruins, also its ideological recon- 
struction as shown in Fig. 2. Wooden church in Smigovec can be virtually dis- 
mantled according to individual layers (roof, plaster, tower, tiling...). The railway 
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viaduct in HanuSovce nad Toplou is complemented by an animated 3D model of a 
historic locomotive and can be let out, which is a reference to the historical event of 
the end of World War II. The water mill in Vy8na Jablonka is currently located in 
the Humenné open-air museum, where the source of flowing water is not possible, 
so this was added in the virtual space and at the same time the animation of the 
rotation of the water wheel was realized. The church in the Vel’ka Tia can be seen 
in three different states, given the historical development of its appearance in dif- 
ferent periods. Some other objects are added to some objects. For the Mill from 
Vy&sna Jablonka there is added a wall with a picture of the building layers, for the 
Wine Cellars in the Velka Tifa are added 3D models of barrels for wine 
production. 


3.1 Virtual Reality Application 


When designing a virtual reality application, we suggested that we do not want to 
offer a full-featured application with storytelling to students, as we assumed the 
involvement of the teacher in the application. Although the application is not col- 
laborative and is designed for a single user, it is possible to display each scene for the 
objects separately, which also gives the teacher the opportunity to concentrate the 
students’ attention on the various details in the scene. Within the scene, you can track 
the location of the user in the space that was specified when calibrating the headset. 
However, the scale of individual scenes is significantly larger, and since we offer 1:1 
object display, it is necessary to ensure that the user moves both in the scene and 
outside the reserved area. For movement of the user in the scene, we chose the ability 
to jump positioning through the Oculus touch controller or HTC VIVE controllers 
through pointing to the terrain. For most objects, the camera in the virtual space is 
inserted at the level of the average human height above the terrain. To view the ruins 
of the Zborov and Nevice castles, we placed the camera so that the user sees the 
objects from the viewpoint, and when changing the position to the terrain, he gets to 
the appropriate place at a level of about 20 m (in the scene units) above the terrain. 
We have chosen this approach to test user responses in different scenarios. 

An important part of the scenes are panoramas, whether in the form of interior 
panoramas, or for some scenes, the panoramas are the main view of the content. As 
part of the design of the system, we have solved the dilemma whether it is necessary 
to create stereoscopic panoramas, or it will be enough to use the classic panorama 
without the possibility of perceiving the depth. As it turned out during testing, it is 
enough for users to use panoramas without stereo, because when changing the 
scene between the exterior (3D model) and the interior, users do not see the change 
between views as a major problem. We used hotspots to move between the scene 
and the spherical panoramas used in the app. As it turned out when testing the app, 
users, especially younger people, did not have the problem of being aware of the 
change in the view and the way of navigation in the 3D scene and in panoramas 
where we had a limited possibility of movement. 
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Romansko-goticky kostol Velka Trna 


Fig. 3. Position of the description text above the controller 


Additional information about the objects in the scene is displayed with a text 
description. Their display is realized after the controller has been directed to the text 
appears above the controller description of the part of the object so that the position 
of this description is linked to the position of the controller itself, as you can see in 
Fig. 3. 


3.2 Augmented Reality Application 


For apps with augmented reality, we’ve been forced to optimize dramatically 
because they are triggered on mobile devices that do not have such high perfor- 
mance, and there is no need to enter all the data for that scene, because in the 
extended reality, users focus only on that object. That’s why we’ve reduced the 
terrain and eliminated the vegetation. Because the entire content of the application 
is stored locally and not dynamically loaded, we also optimized the size of the 
textures and the shaders used. 

The individual scenes are displayed in the application by using the image pattern 
for tracking the space and determining the global coordinate system of the scene. 
3D scenes are displayed above the tracked image without scaling, so scaling is 
controlled only by changing the device’s distance from the tracked image. Testing 
has shown that this method is most user-friendly. Figure 4 shows the scene in the 
application. 

You can view panoramas in the augmented reality application by clicking on the 
hotspot in the 3D scene. The panoramas themselves are displayed on the entire area 
of the device and do not use a tracked image to determine their location in the room, 
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Fig. 4 3D model of ideological reconstruction and ruins of Zborov castle in the augmented reality 
application 


but they can be viewed using the internal sensors of the mobile devices (ac- 
celerometer, gyroscope, magnetometer). As the user rotate, the angle of view 
changes to part of the panorama. During the development of the application we also 
tested the possibilities of linking the panorama position with the tracked image so 
that the user did not lose the context. For these needs, we mapped panoramas on an 
inverse sphere, the centre of which was above the tracked image. As it turned out, it 
was not possible to display all parts of the panorama in this way. 


4 Results 


Utilizing our application for virtual and augmented reality in teaching requires 
teacher guidance. Teacher provides students with information that is complemented 
by visual and audio channels through 3D scenes. Within the framework of the 
implemented project, working methodological sheets were created, which the 
selected teachers directly participated in. The working methodological sheets are 
conceived in five cross-cutting themes (civil society, traditional folk culture, his- 
tory, religion, and ICT) that complement each of the created applications. The 
individual activities in the working methodological sheets are adapted to accom- 
pany students using storytelling to obtain expert information on historical, geo- 
graphic and factual contexts. In particular, the augmented reality application is used 
for group activities, and for individual activities, they are applications using virtual 
reality. 
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4.1 Mixing VR and AR Together in Education 


Within the InovEduc project, applications for virtual and augmented reality have 
been developed to enable them to engage in the learning process individually or 
collectively, and at the same time that students as the primary target group of the 
project are able to use them at home as well. Within the app, the same content is 
offered in spite of a different approach within the interaction and display modes. 
With this approach, it is possible to combine different approaches directly in the 
classroom, independent student interaction in a virtual reality without the teacher 
having to enter the process or the augmented reality use of the group with the 
teacher’s commentary and the possibility of storytelling. Optionally, separate use of 
home-based applications, depending on the HW available to the user. 

Based on feedback from teachers, such a combination of individual approaches 
appears to be an appropriate complement to traditional approaches to learning, 
while the use of multiple application types allows for greater variability in the 
selection of different learning variants along with methodological worksheets for 
the individual subjects being taught. The impact of selected technologies on 
learning experience can be measured individually by the psychological factors, that 
is, presence, motivation, cognitive benefits, control and active learning as men- 
tioned in Lee, Wong and Fung (2010). 


4.2 Testing of the Memorability of the Information 
by Using AR/VR 


An important indicator for education is how to memorize information and how long 
it is stored in the memory. Therefore, when measuring the impact of using our 
applications on the teaching process, we have decided to measure just how much 
information students are able to remember about digitised cultural heritage sites. In 
the process of remembering, an important factor is the induction of emotion that 
improves the way the information can be memorized, regardless of whether the 
emotion is positive or negative. Virtual reality is something new to many users, and 
immersion itself gives them a positive emotion. In the impact measurement, we 
created two groups of students. The task of each group was to remember as much of 
the factual data about the objects. Group A (26 students) worked in a traditional 
way without the use of VR and AR, and Group B (24 students) used the appli- 
cations in the classroom. Subsequently, we created a standardized 20-question test 
and the students had to answer them correctly. This test was carried out immedi- 
ately after the end of the lesson, followed by one week and one month. The 
measurement results are shown in Table 2. 

The results processed in the graph can be seen in Fig. 5. 

The results have shown us that the use of virtual and augmented reality methods 
is not only a fun way of submitting information but it also affects the memorability 
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Table 2 Measures of the impact of applications to memorizing facts about objects 


Group % of correct answers % of correct answers % of correct answers 
immediately after 1 week after | month 

A—without 90 76 52 

VR/AR 


B—with 95 83 71 
VR/AR 


100% 
90% 
80% 
70% 
60% 
50% 
40% 
30% 
20% 
10% 
0% 
% ofcorrectanswers % of correct answers after % of correct answers after 
immediatelly 1 week 1 month 


—@—A-—without VR/AR —@B-— with VR/AR 


Fig. 5 Graph of the result of the testing 


of the information obtained. On the other hand, the biggest obstacles to technology 
deployment to schools have shown problems with hardware equipment, but in 
connection with applications for virtual reality and the problem of motion sickness 
(manifested in about 12% of the tested persons), despite the fact that user tracking 
in the area by external sensors. 


4.3 Applications in the InovEduc Project 


The selection of individual cultural heritage objects that were processed into scenes 
in the InovEduc project was made so that scenes can be used universally not only 
for education but also for the propagation of these objects on-site or off-site. It was 
important to create apps that use a variety of technologies to display content, so 
we’ve created virtual reality apps, mobile apps for augmented reality, but also 
standalone and web apps that can show content without the use of special hardware, 
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and it’s possible also be used for presentation to a wider audience, not just an 
individual user. 


5 Conclusions 


As we have seen in the article, virtual and augmented reality can positively influ- 
ence the memorability of the information obtained within the learning process. The 
knowledge gained through this approach is kept longer in the user’s memory, 
assuming that this fact is significantly influenced by the so-called wow effect, when 
the positive or negative emotion triggered results in the storage of information in 
the long-term memory as stated by Tyng, Amin, Saad and Malik (2017). Verifying 
fact, that this is caused by virtual and augmented reality we anticipate as a future 
work in collaboration with psychologists. The created applications can be used 
directly in the teaching process, but also in home education. 

Moreover, due to the fact that the content itself has been chosen as cultural 
heritage, other possibilities of utilization such as promotion in tourism and the like 
are also open. Thanks to the versatility of the processed content and the individual 
applications created, it is possible to mediate the content on different technology 
platforms (web, touch panels...) and in different places (on-site or off-site) in 
various educational, cultural and self-government institutions. In addition, digital 
technologies can bring new experiences in the visualization of events that cannot be 
repeated in the real space (for example, the destruction of the railway viaduct in 
HanuSovce nad Topfou). 

Despite the fact that there are still many obstacles to a larger boom in tech- 
nology, there is evidence of the use of virtual and augmented reality as a good step 
towards improving the learning process. As stated by Kriz (2017), such learning 
methods can also be used to reduce barriers between countries. The created 
applications have the potential for their further development, for example, towards 
the processing of other topics, such as the history of local transport, mentioned in 
the article of Stefanovié and Schindler (2016). 
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Part V 
AR & VR Applications and 
Immersive Designs 


To Have and Vehold: Marrying Mm) 
Museum Objects and Virtual a 
Collections via AR 


Ronald Haynes 


Abstract The Veholder.org project is aiming for enhanced, innovative museum 
exhibitions, enabled by collaborative use of Augmented Reality. Complementing 
physical with relevant 3D virtual objects widens impact of interrelated collections. 
Objects brought together, from display or storage, provide blended environments 
for visitors and researchers. Veholder (Virtual Beholder, or Virtual Environment for 
Holdings and Online Digital Educational Repositories) is developing collaborations 
between suitable institutions. Early tests at the University of Cambridge are 
promising, while challenges such as calibration and scaling stress the need for 
shared solutions to blend collections. Following initial discussions with developers 
of INF, the International Image Interoperability Framework, about standards for 
digital library sharing of historical manuscripts and other disparate 2D images, we 
hope that working together can accelerate the process for a standardised approach to 
sharing 3D images. This could extend the concept of universal digital library 
viewers to incorporate and integrate 3D and AR images as well. 


Keywords Museums - Virtual collections - Blended collections + 360 viewing - 
3D scanning - Calibration and real-world scaling 


1 Introduction 


A visit to a museum is like immersion in an illustrated book, where key concepts 
have emerged from the textual contents and taken on full form, and suddenly 
sprung into vibrant full shapes. What we read in books will often draw us to a 
museum, to see in embodied form the ideas and items described, or to experience 
more of the material culture and cabinet curiosities which provide content and 
context for our tales. Conversely, the museum will often suggest further reading and 
follow up literature to carry forward ideas inspired by and presented in collection 
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displays. This interrelation of the primary 2D and 3D aspects of our communica- 
tions, culture and collections are perhaps the earliest and most extensive clues of 
what we know about our ancient human ancestors and species relatives. Early 
humans combined what they crafted with what they collected and conveyed this 
through drawing and construction, through various means of story-telling, visual 
representation, and writing. 

The Veholder.org project has been working with institutional groups interested 
in exploring and addressing some challenges with expanding from 2D into 3D 
imaging, to enable and improve collaborative Augmented Reality and with other 
museums collections. The project was introduced at the 3rd International AR and 
VR Conference held 23rd February 2017 in Manchester (www.mmu.ac.uk/ 
creativear/conferences/2017-augmented-and-virtual-reality-conference-2017 [June 
19, 2018]), as described in “Eye of the Veholder: AR Extending and Blending of 
Museum Objects and Virtual Collections” (link.springer.com/chapter/10.1007% 
2F978-3-319-64027-3_6 [June 19, 2018]). Veholder is a term intended to creatively 
describe someone who, by virtue of the ability of AR to wed physical and com- 
plementary virtual items, becomes a “Virtual Beholder”. Veholder also was formed 
as the acronym for “Virtual Environment for Holdings and Online Digital 
Educational Repositories”. 

Although some museums may have previously been reluctant to scan and share 
objects, with concerns about reducing visits to the museum, the Veholder project 
represents an AR extension of the idea of special exhibitions and the 
well-established practice of lending objects from one collection to another. Such 
special exhibitions regularly increase a museum’s attendance, however they also 
can provide the perfect opportunity to scan and share for future agreed use by the 
already collaborating museums. In a special event for its Members, the Director of 
the British Museum hosted the Director of the State Hermitage Museum (St. 
Petersburg) in a discussion about the role of the encyclopaedic (or universal) 
museum in the 21st century (www.britishmuseum.org/whats_on/events_calendar/ 
event_detail.aspx?eventId=3906 [June 19, 2018]), as well as the future of museums, 
their roles and options for collaboration. It was noted that one of the Parthenon 
Sculptures had been lent to the Hermitage for a special exhibition, as part of their 
250th anniversary. Although great lengths were taken during transit to avoid 
controversy with Greece over the location of the sculptures, such concerns raise 
questions of combining casting and 3D scanning and printing, along with AR 
imaging options—concerns which the Veholder project aims to help address. 

The project has been developing potential collaborations with suitable groups in 
Cambridge, where there are multiple museums (www.museums.cam.ac.uk [June 
19, 2018]) and research collections, with a project proposal submitted in con- 
junction with the University of Technology Sydney, for the forthcoming 250th 
anniversary of Cook’s landing at Botany Bay, proposing 3D images and duplicates 
of expedition artefacts in Cambridge and London. There also have been partnership 
discussions with the Natural History Museum of Denmark (snm.ku.dk/english 
[June 19, 2018]), University of Copenhagen, and Cambridge’s Museum of Zoology 
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(www.museum.zoo.cam.ac.uk [June 19, 2018]), and Duckworth Collections 
(www.human-evol.cam.ac.uk/duckworth.html [June 19, 2018])}—potentially working 
together on a pan-primate 3D catalogue. 

More recently, we have had initial and promising discussions with a network of 
developers and implementers of IIIF, the International Image Interoperability 
Framework (iiif.io [June 19, 2018]), about the interoperability standards they have 
advanced for the development of digital libraries. We hope that working together, 
including with the IMF 3D community group (iiif.io/community/groups/3d [June 
19, 2018]), can accelerate the process for a standardised approach to sharing 3D 
images, as well. This could extend the concept of universal digital library viewers, 
which they have created and are promulgating, to incorporate and integrate 3D 
images and ideally AR techniques as well. 


2 Issues 


While the aims of the Veholder.org project include enabling enhanced AR col- 
laboration between museum holdings, archives and collections, the focus has lar- 
gely been on images, display and visual technologies. For the future and for more 
immersive experiences, it is important to consider the wider scope of AR devel- 
opments, which are attempting to engage us via other senses, in particular through 
smell, touch, and sound. For example, one approach to AR via the sense of smell is 
called Cyrano, by oNotes (onotes.com [June 19, 2018]) which is capable of pro- 
grammable scent scenarios which can be shared between units, and which the 
producers refer to as the “first digital scent player”. 

Touch is perhaps even more surprising for AR, and a new prototype tablet called 
Tanvas (tanvas.co [June 20, 2018]) simulates the feeling of various textures, 
including choppy, grainy, fine, and wavy (see Fig. 1). The combination of feeling 
connected with what we are seeing on a screen could be especially powerful for 
AR, and particularly helpful for interaction with virtual objects, providing a richer 
experience of the physical characteristics of 3D objects than available to sight 
alone. 

The image (Fig. 1) is of Tanvas, a prototype touchscreen which simulates how 
things feel, to augment what you see on the screen. Using electrostatic fields to 
create friction, through touch it can convey textures such as choppy, grainy, fine, 
and wavy. (ohnbiehler.com/2017/01/11/wired-wednesday-favourites-ces-2017 
[July 21, 2018]) 

Smart glasses are being developed with new sensory options for AR, as are 
tablets, with prototypes available for providing AR with hearing. In particular, Bose 
smart glasses (www.theverge.com/2018/3/12/17106688 [June 19, 2018]) look like 
sporty sunglasses, but can be paired with a smartphone to supply directions and 
commentary about where you are and, to some extent, what you are seeing. They 
can also deliver great music, channelling sound into the ears without using in-ear 
plugs. Other audio equipment companies seem interested in this area, and there 
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Fig. 1 Image courtesy of John Biehler 


should be some combinations of such smart audio with visual technologies for even 
smarter smart glasses. 

Generally, however, there continue to be more developments in the area of sight 
and displays, including new options for smartphones and smart glasses. And yet, as 
with other areas of innovation we have a connected challenge of divergent direc- 
tions and incompatible systems, as well as initially producing technologies focussed 
almost exclusively on solitary experiences. Thankfully, there are increasing efforts 
and indications of interest in greater collaboration and commitment to evolving 
standards. 

From the early advances with virtual reality technologies in the mid 1800s, with 
the explosive expansion of stereographs and viewers, what was crucial to the 
widespread success was the early setting of a standard. In general, any of the many 
viewers available could be used to view the hundreds of thousands of stereograph 
cards produced (see Fig. 2). Those inclined could construct their own viewer, as 
well. Then, as now, a standard ensured that the viewer and cards would work 
together, no matter their origin. This legacy has endured, and stereographs from the 
Victorian era still work as designed and pleasingly well on modern VR and AR 
systems. 

The first image (Fig. 2) shows Sherlock Holmes (Ronald Howard, 1954 TV 
series, Episode 28) using a stereoscope (archive.org/details/SherlockHolmes 1954 
[June 19, 2018]). The enormous popularity of stereographs prompted development 
of options for group sharing. The second image (Fig. 2) is a Kaiserpanorama 
(www.stadtmuseum.de/ausstellungen/kaiserpanorama [July 22, 2018]), also known 
as a Fotoplastikon, a Victorian era innovation for communal 3D viewing, precursor 
to cinema, and still in selective use. 
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Fig. 2. Images courtesy of Guild Films/Heiko Noack, Stadtmuseum Berlin 


While smart glasses for AR use continue to evolve, many projects have been 
developing around Microsoft Hololens (www.microsoft.com/en-gb/hololens [July 
21, 2018]), at the same time that there has been extensive coverage and planning 
around the well-funded and promising Magic Leap (www.magicleap.com [July 21, 
2018]), due for release in 2018. Epson’s Moverio glasses (www.epson.co.uk/ 
moverio [July 21, 2018]) have a third generation model designed for multi-person 
use, particularly in museums (BT-350), as well as a novel release of the 
multi-person display and frames which can be used to develop or deploy with other 
smartphones or computer systems (BT-35E). 

Among concerns expressed by a number of museums, about AR equipment and 
other technologies, are questions about affordability, robustness and sustainability. 
This is why some still prefer to focus on smartphones alone, with the idea that 
interested visitors can bring their own devices and use a suitably prepared 
app. Others have been hoping to pair up visitors’ smartphones with the AR equiv- 
alent of Google Cardboard (vr.google.com/cardboard ([July 21, 2018]). Late 2017 
and early 2018 saw some innovators in this area, with two leading examples (see 
Fig. 3) in the Aryzon AR Headset (www.aryzon.com [June 19, 2018]) and the Haori 
Mirror (haoritechnology.com/en/col.jsp?id=109 [June 19, 2018])—or Docooler AR 
Headset, in the UK (www.amazon.co.uk/dp/B07851GG8Q [June 19, 2018]). 


Fig. 3. Images courtesy of Shenzhen Haori Technology Co. Ltd./Aryzon B.V. 
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The first image (Fig. 3) is of the Haori Mirror (marketed in the UK as 
Docooler AR Headset), with its Bluetooth controller. The second image (Fig. 3) is 
the Aryzon headset kit. Each has their own apps, can use a wide range of smartphones 
for viewing, works with a variety of AR and VR resources, and for AR can use an 
included target to place and anchor an image in local space. Each retails under £50. 

As a potentially fruitful complement to smart glasses, or smartphones in headset 
holders or on their own, there are some recent options available for transparent 
touchscreen computer displays which can be built into museum cases. These can 
achieve some of the AR effects without glasses, directly shareable with groups, 
although of course within a fixed location rather than in a portable mode (see Fig. 4). 

The images (Fig. 4) are from a demonstration of an interactive transparent 
display, using a computer touchscreen forming the main window for exhibiting a 
book within the cabinet (youtu.be/OeRpeBchZOs [June 19, 2018]). 

While new VR goggles and AR glasses (and novel displays) continue to emerge, 
along with new and alluring features, there continues to be sufficiently diverse and 
proprietary platforms to make it difficult to develop for the many format and feature 
differences. There are helpful approaches to these cross-platform challenges, 
notably with the widespread developments using the Unity engine (unity3d.com 
[July 21, 2018]). Yet, there have been understandable concerns about the lack of 
common ground and of standards, including variations in the use of the terms VR, 
AR, MR and XR (https://www.forbes.com/sites/charliefink/2017/10/20/war-of- 
arvrmrxr-words [June, 2018]), which have meant confusion as well as often 
higher costs of development for those who want to publish VR or AR (or related) 
content across the many available platforms. 

In response, there are recent and high-level movements toward open standards, 
for development and delivery to be via our web browsers, with Mozilla leading the 
push for WebXR (blog.mozilla.org/blog/2017/10/20/bringing-mixed-reality-web 
[July 21, 2018]), built on earlier open standards in order to provide a common 
programming interface to simply development for both AR and VR devices. 
Mozilla is also providing the Firefox Reality browser (blog.mozilla.org/blog/2018/ 
04/03/mozilla-brings-firefox-augmented-virtual-reality [July 21, 2018]), designed 
to help ensure these standards are available for stand-alone VR and AR headsets. 
WebXR is also being supported by Google (www.vrfocus.com/2018/05/google- 
introduce-webxr-standard-to-chrome July 21, 2018]) and Amazon (www.zdnet. 
com/article/aws-sumerian-a-bet-that-enterprise-augmented-and-virtual-reality-will- 
be-browser-based [July 21, 2018]), and the combined commitments should ensure 
continued traction for these much-needed development and delivery standards. 


3 Procedures 


To demonstrate the great potential for AR delivered via smartphones, and to pro- 
vide initial samples of what options for AR with museum collections might be like, 
some photographic experiments were carried out to situate and display 3D models 
in selected physical spaces to produce suitably combined images. 
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Fig. 4 Images courtesy of Crystal Display Systems Ltd. 


The 3D models were selected from among the vast offerings found on Sketchfab 
(sketchfab.com [June 19, 2018]). The models were displayed using the Sketchfab 
app in AR mode on a suitable smartphone, with these experiments carried out using 
a Samsung Galaxy S8. Appropriate physical settings were selected to highlight the 
AR possibilities. 

With 3D models of scans taken from Cambridge and British Museum collec- 
tions, below are selected representatives of the initial outcomes (see Fig. 5). 


4 Results 


Following the promising initial samples, the AR photographic experiments pro- 
gressed to bring together a pair of similar physical and virtual objects, and 
Sketchfab was again a good source of models, this time from Oxford and further 
afield (see Fig. 6). 

The image (Fig. 6) shows an AR combination of a physical-virtual composite, 
where the skull on the right is a 3D model of Australopithecus afarensis (Lucy), 
from the Oxford Natural History Museum collection (skfb.ly/HyJs [June 19, 2018]). 
The 3D model was scaled and positioned on the physical shelf, alongside the 


Fig. 5 Images courtesy of Ronald Haynes/3D models courtesy of Sketchfab. The first image 
shows a Neanderthal skull model from Cambridge ( https://skfb.ly/6yTt9 [June 19, 2018] visually 
scaled and placed on a tabletop, to produce a virtual-physical composite as an AR test. The second 
image shows a Cuneiform tablet model from Cambridge (skfb.ly/PySr [June 19, 2018]) above the 
same tabletop. The third image is of the same tabletop with an Easter Island monolith model from 
the British Museum (skfb.ly/6srQY [June 19, 2018]) 
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Fig. 7 Images courtesy of Ronald Haynes/3D model courtesy of Sketchfab 


physical skull on the left, a modern human skull from the Duckworth Collection in 
Cambridge (www.human-evol.cam.ac.uk/duckworth.html [June 19, 2018]). 

The side-by-side AR skulls experiments were well received and encouraging for 
additional testing. A next logical step was to attempt to virtually place a suitable 3D 
model among the physical objects within an existing museum case. A suitable case 
and objects were located, and once again Sketchfab was a good source of a 3D 
model, this time a computer-generated one from an independent project (see 
Fig. 7). 

The first image (Fig. 7) shows a display case before any AR experimentation, 
and in the left-hand corner is a large zoetrope and image strips designed by the 
physicist James Clerk Maxwell, key apparatus for what is believed to be the first 
time a moving picture was used for scientific demonstration. The case is part of the 
Cavendish Laboratory museum collection, in Cambridge (www.phy.cam.ac.uk/ 
outreach/museum [June 19, 2018]). The second and third images include a 3D 
model of a praxinoscope (skfb.ly/6pQqK [June 19, 2018]), which is scaled and 
positioned to the right and looking nearly at home alongside its relative the 
zoetrope. 
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5 Conclusions 


Along with the very promising results of the AR image tests noted, there remain 
concerns about scaling and image standards. It has been most encouraging to find 
the great progress which has been made in the digital library and archive world, 
dealing with the challenges of 2D imaging, including how to address the many 
difficulties associated with fragmented manuscripts. After meeting developers and 
community leaders from IIIF, the International Image Interoperability Framework 
community (itif.io [June 19, 2018]), it has been very helpful to learn more about 
their progress and practices. IIIF has defined a set of programming interfaces, based 
on open web standards, derived from shared use cases, and is also is the community 
that implements the specifications. 

IF has had a particular concern for the difficulties surrounding the sharing of 
historic manuscripts and other often disrupted 2D texts and images, whether due to 
fragmentation of the originals, or otherwise torn, worn or missing pieces, which at 
times can be reconstructed by virtually putting the pieces back together. For these 
challenges, they have created standards for interoperability, and introduced the 
concept of universal viewers, which can present composite images assembled from 
local sources and remote links to collaborating systems which are following the IIIF 
standards and so are compatible. The resulting environment ensures digital library 


Fig. 8 Image courtesy of Digirati 
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patrons and researchers can reunite or creatively assemble more of the worlds 
disconnected knowledge (see Fig. 4). 

The image (Fig. 8) is an illustration of how INF can help reunite the fragmented 
and distributed parts of a manuscript (resources.digirati.com/1iif/an-introduction- 
to-iiif [June 19, 2018]), where existing pieces are presented in the interoperability 
framework. 

It also has been very heartening to find the TIF 3D interest group (iiif.io/ 
community/groups/3d [June 19, 2018]) and the open and collaborative approach 
they are taking to clarify interoperability and other challenges with 3D imaging. 
There is hope that some of the great success with interoperability and universality in 
sharing 2D materials, in particular through a viewer which can connect and inte- 
grate disparate resources, will help guide a similar process for 3D imaging. It is a 
worth considering ways to be able to bring together digital texts and images, and 
3D models in one environment, for instance to enable with one viewer the ability to 
review da Vinci’s notes, to view his illustrations, and to interact with 3D models 
based on his designs. Similarly, although Darwin’s publications and manuscripts 
are online, the specimens from the Beagle expedition are divided between muse- 
ums. The same integration in a universal viewer would benefit these and any other 
areas where image and text are meaningfully connected with 3D objects, and vice 
versa. 

The option to further test and potentially incorporate high-feature smartphones, 
rather than solely focus on one or more proprietary smart glass models, opens up 
many more options for the always budget-sensitive museums. The move toward 
open web standards and increasing adoption of WebXR will also pave the way for 
greater collaboration, as well as more flexible and sustainable projects. It is hoped 
that the work with INF will also help advance efforts toward standards in 3D 
imaging, scaling and interoperability, to simplify sustainable collaborations 
between institutions. 

The Veholder project may be best developed in optional phases, to start testing 
the technology at the soonest, and help build the required collaborations, including: 


e Phase I—360-degree live-streaming guided option, with guides to introduce 
new technology and help blend collection images and concepts. 

e Phase II—introduction of 3D scanned images, with guides to introduce the 
technologies and collections, clarifying any oddities. 

e Phase [[—initial curated, blended special exhibition, with 3D images suitably 
scanned and scaled for compatibility across collections. 

e Phase [V—development of a larger catalogue of suitably matched 3D images, 
available for more extensive combinations across collections. 


More details of the Veholder project will be found on www.veholder.org. 
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A Tool, not a Toy: Using Virtual Reality ®) 
to Evaluate the Communication Between poet tas 
Autonomous Vehicles and Pedestrians 


Sebastian Stadler, Henriette Cornet, Tatiana Novaes Theoto 
and Fritz Frenkler 


Abstract Although the main market for Virtual Reality (VR) is currently the 
gaming industry, advantages of using virtual environments in research and devel- 
opment have been already demonstrated e.g. for car industry or urban planning. 
Especially when no prototype is feasible or available, VR constitutes an advanta- 
geous alternative since it allows tests in laboratory conditions with high flexibility 
and ensured safety for test participants. In the presented study, it is investigated how 
VR can be used as a tool for Usability Tests to evaluate Human Machine Interfaces 
(HMI) for communication between autonomous vehicles and pedestrians. Singapore 
with its regulations and requirements has been selected as reference. Beyond the 
findings that explicit HMI concepts improve the communication between autono- 
mous vehicles and pedestrians, VR was validated as suitable tool to conduct 
Usability Tests. Further studies plan to integrate additional case studies as well as 
improved immersion of test participants within the virtual environment. 


Keywords Virtual reality - Usability tests - Human machine interfaces - 
Autonomous mobility 
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1 Introduction 


Due to the lack of a driver in level 5 autonomous vehicles’ (AV), communication 
between AVs and human road users, like pedestrians, has to be replaced. Human 
Machine Interfaces (HMI)—for instance, on-vehicle screens with real-time infor- 
mation—are a possible solution (Matthews & Chowdary, 2017). The critical nature 
of the task (i.e. crossing the street when an AV is approaching) demands visible and 
comprehensible information in order to ensure pedestrians’ safety. Especially in a 
multicultural environment like Singapore,” universal comprehensibility of written 
information and/or symbols, which constitute the HMI concepts, is a prerequisite. 
Therefore, the usability of HMI concepts must be tested, as the implementation of 
such communication without appropriate verification could lead to misinterpreta- 
tion and thus safety hazards. 

However, the autonomous mobility context brings significant obstacles to reli- 
able validation of HMI concepts mainly regarding complexity, effort, and safety. 
Indeed, in order to create an authentic traffic condition, a physical test bed has to be 
set up, including e.g. roads, junctions, sidewalks, traffic lights, zebra crossings, and 
traffic signs. Additionally, both manually driven cars and AVs have to be integrated 
into the test bed. Test participants, which include drivers, pedestrians, and pas- 
sengers, also have to be present to create the environment. To ensure a reliable data 
collection, all these aspects have to work together perfectly—which results in a 
complex endeavour and leads to great effort regarding time and money spent. 
Furthermore, AVs raise concerns about the technology’s safety (Eng, 2017). 
Misinterpretation can lead to accidents, as it became visible at the fatal crash caused 
by a Tesla in self-driving mode (The New York Times, 2016). Therefore, testing 
the usability of communication between AVs and pedestrians in real-life conditions 
remains potentially dangerous. 

The hypothesis of the presented study is that Virtual Reality (VR) is a suitable 
tool to test the usability of HMI concepts between AVs and pedestrians as a 
replacement for tests in real-life conditions. Therefore, the objective is to develop a 
suitable methodology for Usability Tests within VR in the context of autonomous 
mobility. 


‘In contrast to level 0 automation, which means the human driver has to perform all aspects of the 
driving task, level 5 automation means that humans do not overtake or influence any task in any 
driving situation, but act solely as passengers (SAE International, 2016). 

*For the investigation of communication between AVs and human road users, a geographical 
context is required. Singapore has been selected for this study. Its local regulations and cultural 
environment will be considered as requirements. 
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2 Literature Review 


2.1 Context: Existing HMI for Communication 
Between AV and Pedestrians 


The focus of this paper lies on communication between AVs and pedestrians, in an 
ambiguous situation at a one-way street without traffic lights and/or zebra crossing 
and a potential intention of crossing the road by the pedestrian while an AV is 
approaching. For this type of communication, a distinction can be made between 
explicit and implicit communication. While explicit communication implies direct 
messages exchanged between the road users (e.g. light signals, horn, gestures), 
implicit communication is linked to indirect messages in which the content is not 
directly addressed (e.g. a car reduces its speed to encourage pedestrians to cross) 
(Fuest, Sorokin, Bellem, & Bengler, 2017). Explicit communication occurs between 
pedestrians and drivers mainly via gaze and/or gestures (Sucha, 2014). Without 
driver in an AV, this communication has to be provided alternatively in the future. 

Multiple car manufacturers proposed HMI solutions for their Level 4 and Level 
5 AV concepts, like for instance the Mercedes F 015 (Mercedes, 2015) or the 
Nissan IDS concept (Nissan, 2015). In order to deal with the missing communi- 
cation between drivers and pedestrians, on-vehicle displays and/or projection 
technology are possible solutions to signalize for instance instructions to the 
pedestrians. 

However, comprehensive testing and validation of the results are not published. 
Some researchers—as Clamann, Aubert, and Cummings (2017) and Benderius, 
Berger, and Lundgren (2017)—have proposed and tested concepts by using regular 
cars disguised as autonomous vehicles, with mixed results regarding the interfaces’ 
effectiveness. 

With these insights, a method, which is safe and easy to set up, for evaluating the 
usability of communication concepts between AVs and pedestrians is still lacking. 


2.2 Testing Usability 


Usability is defined as ‘the degree to which something is able or fit to be used’ 
(Oxford Dictionary, 2018). Rubin and Chisnell (2008) define a product or service 
as truly usable when the user ‘can do what he or she wants to do the way he or she 
expects to be able to do it, without hindrance, hesitation, or questions’ and speak of 
‘absence of frustration in using something’. van der Bijl-Brouwer (2012) studied the 
link between varying use situations and usability. In the presented context of AV to 
pedestrian communication, it means that the designed HMI should provide infor- 
mation (e.g., intention and/or instructions) that the user needs in order to suc- 
cessfully complete a task—in this case, crossing a street—and that he/she 
understands without hesitation or question. 
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Table 1 Attributes of usability and implications for the study 


Attribute General definition Context of the study: HMI for AV to 
pedestrian communication at a 
one-way street 

Usefulness To which extend a product or service | How should be the HMI designed so 

supports the user to reach his/her goal | that the user is supported in his/her 
with regard to the willingness from the | intention to cross the road? Is an HMI 
user’s side to use the product or even needed? 
service in the first place 
Efficiency Time, accuracy and degree of Is the HMI supporting a faster 
completion to reach the user’s goal decision-making for crossing the 
road? 

Effectiveness | To which extent a product behaves as | Is the HMI adapted to the situation 

expected and the environment e.g. regarding 
traffic conditions and safety? 
To which extent can the HMI prevent 
wrong behaviour from pedestrians? 

Learnability | Ability from user’s side to operate a_ | Are any competences from the user 

product or system considering a side required to understand the HMI 
certain level of competence to operate | correctly? Is a phase of education 
the system after a predefined period of | necessary to understand the message 
time provided? 

Satisfaction | Subjective feelings, perceptions and How does the user perceive the HMI 

opinions from user’s side to reveal for crossing the road? 
users’ satisfaction levels 

Accessibility | Access to the products or services that | Is the HMI understandable for people 

are needed to reach the goal especially | with disabilities e.g. cognitive 
for users with disabilities (e.g. disabilities? 

temporary or permanent limited 

mobility) 


Source Based on Rubin and Chisnell (2008) 


According to Rubin and Chisnell’s definition (2008), to be useful, a product or 
service should enable usefulness, efficiency, effectiveness, learnability, satisfaction, 
and accessibility. Table | presents the definition of these attributes according to 
Rubin and Chisnell as well as their implications for the study. 

For answering the questions raised in Table 1, several methods and techniques 
can be used and combined to evaluate the usability of a product or service such as 
observation of users in real-life environment, surveys, expert interview or classical 
experiments with large sample sizes and control groups. Considering the advan- 
tages and drawbacks of the methods for evaluating usability based on the work of 
Rubin and Chisnell (2008) as well as van der Bijl-Brouwer (2012), Usability 
Testing is selected for the presented study. 

This method provides empirical data from the observation of representative users 
while using a product or system. Compared to classical experiments, Usability 
Tests are more informal, iterative and give qualitative insights of a product’s or 
service’s usability. Basic elements that are included in the method are: 
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(a) The articulation of research questions or test objectives 

(b) A representative amount of users (randomly or not randomly chosen) 
(c) Representation of actual environment 

(d) Observations and Interviews 

(e) Quantitative and qualitative data. 


However, the representation of actual environment for Usability Tests raises 
issues in the presented study of AV to pedestrian communication since the conduct 
of Usability Tests in real-life conditions with AVs and pedestrians is potentially 
dangerous for the test participants. Moreover, building up a testbed in real-life 
conditions to test the AV to pedestrian communication would lead to great effort, as 
well as time and money to be spent. 

To tackle safety issues, costs, and time spent, VR is investigated as alternative to 
Usability Tests in real-life conditions. 


2.3 Virtual Reality 


VR’s major scope of application is the gaming industry. One reason for this is the 
improved immersion into virtual environments and therefore enhanced experiences. 
However, it is also used as a tool in industry fields like automotive, construction 
and military (Berg & Vance, 2017). Deb, Carruth, Sween, Strawdermann, and 
Garrison (2017) used VR to conduct research in the field of pedestrian safety since 
it constitutes a safe alternative for test participants. Furthermore, VR is used as a 
research tool to conduct psychological studies thanks to its capabilities to create 
laboratory conditions for the experiments and its high flexibility to create immersive 
environments (Loomis & Blascovich, 1999). Mihelj, Novak, and BeguS (2014) 
state that VR is used for designing and testing machines and objects, especially 
when they are very expensive (e.g. power plants) or when they are produced in 
large quantities (e.g. cars). 

Consequently, running Usability Tests in Virtual Reality (VR) environments is 
proposed as an alternative to real-life tests with the hypothesis that the attributes of 
usability can be evaluated similarly or even with advantages in regard to real-life 
tests, for instance, due to the possibility to neglect unintentional factors like the 
implicit communication of deceleration. 

Therefore, the hypothesis is: Virtual Reality is a suitable tool to conduct 
Usability Tests with a multicultural selection of participants in order to evaluate the 
most usable HMI concept for the communication between AVs and pedestrians in 
ambiguous situations. 
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3 Method 
3.1 Case Study 


The purpose of the presented case study is to measure the usability of HMI display 
content for the AV to pedestrian scenario: A test person stands on the sidewalk of a 
one-way street with one lane and without traffic light and zebra crossing. The task is 
to cross the road as soon as the test person assesses the traffic situation to be safe. In 
each scenario, an AV that is equipped with an HMI concept approaches the test 
person. The HMI concepts indicate if it is safe for the test person to cross the road 
or not. To achieve this, each HMI concept consists of one “Cross” symbol and one 
“Don’t Cross” symbol. The HMI concepts include the commonly understandable 
red and green colour combination, used at Singapore’s traffic lights, in which green 
is used for indicating that the pedestrian has the right of way (Fig. 1): 


Walking man (a) 
Arrow (b) 
Check (c) 

LED strip (d) 
Traffic light (e) 


A control test is made without any HMI concept in order to compare the HMI 
concepts with the absence of AV to pedestrian communication. 

Since the goal is to evaluate the usability of used HMI concepts when applied in 
the AV to pedestrian scenario and not the display technology itself, a simple 
screen-like surface is positioned in front of the vehicle, where the content is shown 
(Fig. 2). Factors like reflections and brightness are neglected. A further controlled 
variable is the vehicle’s deceleration: to avoid that pedestrians decide to cross based 
on the AV’s kinematic cues, the AV’s deceleration was disregarded. 

In light of Singapore’s speed limit regulations for one lane roads, a speed of 
50 km/h (v = 13.9 m/s) was selected for the approaching vehicle (Land Transport 
Authority, 2017). 

Initially, an appropriate distance from which the AV starts displaying the 
information to the user has been calculated (Distance to Zebra, DTZ), which is the 
product of the factors speed (v) and Time To Collision (TTC) (Eq. 1). TTC 


Fig. 1 Concepts tested for HMI content [The HMI concepts to be tested have been developed 
based on internal workshop sessions and surveys (Theoto, 2018)] 
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Fig. 2 Basic interface 


describes the time it would take for the AV to reach the pedestrian’s path if the 
chosen vehicle speed (v) is kept constant (Schneemann & Gohl, 2016). 


DTZ = v- TTC (1) 


TTC is the sum of the minimum perceptual reaction time of 2.5 s, and the 
average time required to cross a single lane of traffic of 2.7 s (based on Clamann 
et al., 2017) (2.5 s+ 2.7 s =5.2 s). The TTC of 5.2 s is within the critical gap 
acceptance interval as observed by Schmidt and Farber (2009): under 3 s, no 
pedestrian crosses the street, whereas everybody walks with a TTC above 7 s. 

Considering a constant speed of the vehicle and a TTC of 5.2 s, the DTZ has 
been set to 72.2 m (Eq. 2). 


DIZ = v- TIC = 13.9 -5.28=72.2m (2) 

S 
Figure 3 showcases the configuration of the virtual environment. The participant 
position is delimited with a cross on the virtual sidewalk, and as soon as he/she 
faces to the right-hand side, the vehicle appears. Other traffic and road users were 


neglected. This guarantees that all participants face the AV at the same initial spot 
and under the exact same conditions. 


_ Starting point 


Fig. 3. Configuration of virtual environment 
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3.2. Test Participants 


As one aspect of the scenario lies on a multicultural environment, the chosen quota 
sample for the recruitment of test participants reflects Singapore’s ethnic distribu- 
tion. Since there is no ethnic data for the large non-resident population (which 
account for 29.4% of the country’s population, i.e. 1.65 million people), the sample 
is based on the citizens and permanent resident (PR) data (Singapore Statistics, 
2017). Therefore, the ethnic composition of the sample is 53.8% Chinese, 10.6% 
Malay, 5.2% Indian, and 30.4% PRs with other ethnicities. 


3.3 Virtual Reality Hardware Setup 


The Virtual Reality Laboratory consists of an empty space of up to 4.5 m x 4.5 m. 
Tracking devices allow investigating position changes as well as head movement. 
Test persons get immersed in the virtual scenario with help of a Head Mounted 
Display (HMD) (i.e. HTC Vive). Input devices allow the test person to interact with 
the virtual scenario if needed. To deepen the immersion, the test person wears 
noise-cancelling headphones. Additional hardware is located outside the tracked 
area. 


3.4 Adaptation of Usability Tests Within VR 


The Usability Tests within VR focus on the following attributes of usability listed in 
Table 1: efficiency, effectiveness, and satisfaction. The attributes are chosen due to 
the fact that the quantitative measures of efficiency and effectiveness (i.e. reaction 
time and error rate) can be collected very accurately with the help of VR. 
Satisfaction is chosen in order to get qualitative insights about the test participants 
feelings and perceptions regarding the HMI concepts. 

For the assessment of efficiency of selected HMI concepts, reaction times are 
measured and compared afterwards with the control group. The reaction time is 
defined as the time it takes from the moment when the test participant sees the AV 
until he/she starts to cross the road. Effectiveness can be derived from an error 
analysis, in which the amount and type or errors (e.g., the pedestrian crosses when 
he/she must not) are collected and analysed. The level of satisfaction is assessed 
with qualitative data, collected with questionnaires before and after the Usability 
Tests. 

The procedure for the tests is shown in Table 2. 

One test person is tested at a time. Prior to the tests, a first questionnaire is 
conducted. Then, the participants are introduced to the VR setup with help of a 
tutorial in order to get familiarized with the technology. After the tutorial, the 
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Table 2 Procedure of the tests within VR 


Procedure 


Tutorial Usability tests Final questionnaire 
2 10 eS) 


First questionnaire 
5 


Activity 


Duration (min) 


Usability Tests are conducted. Here, the test participant gets the task to cross the 
street while an AV is approaching. Since there are five HMI concepts that include 
one symbol for “Cross” and one symbol for “Don’t Cross”, as well as one control 
group, the test is conducted eleven times per test participant. First of all, the control 
group without HMI concept is conducted, followed by the five HMI concepts. The 
sequence of procedure for the HMI concepts is randomized to rule out distorted 
results caused by the testing order. After the tests, a final questionnaire is conducted 
in order to get insight into the feelings and perceptions of the test participants. 


4 Findings 


Overall 18 people participated in the Usability Tests. The ethnic distribution of test 
participants was: 10 Chinese participants, 2 Malay participants, | Indian participant 
and 5 Participants with other ethnicities. 

Figure 4 presents results of the test, i.e. the average decision times for each HMI 
concept. 

The result showed that the decision times differed significantly between the 
control group and any HMI concept. While the average reaction time for the control 
group is 4.8 s, the average reaction times for the HMI concepts lie between 2.0 and 
3.0 s. 

This data proved that the symbolic representations lead to a reduced pedestrians’ 
reaction time. When HMI concepts were not present, the mean reaction time was 
significantly larger (4.8 s instead of 2.0-3.0 s), showing that display intention or 
instructions to pedestrians can help their decision process, at least when no other 
intention indicators (deceleration, engine sound) are present. However, no signifi- 
cant decision time variation among the different HMI concepts was observed. 

Through the collection of error rates, the HMI concepts’ effectiveness regarding 
usability has been evaluated. As Table 3 shows, the test revealed that in 72.2% of 
the tests the control group led to errors, as test participants crossed the street when 
the vehicle exercised its right of way and did not stop for the test participants. When 
the HMI concepts were present, errors occurred only in three out of 90 trials. 

This outcome is an indicator of the HMI concepts’ effectiveness, as the error rate 
declined steeply when HMI concepts were present. A comparison among the dif- 
ferent HMI concepts was possible, as the “Check” and “LED Strip” were less 
effective than the other ones. 
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ini 


=sControl @WalkingMan #Arrow @Check OLED ofraffic 


Fig. 4 Comparison between average decision times, in seconds, for the control group and 
different HMI concepts 


Table 3 Error analysis for HMI concept Bact Hequency @ 
control group and different i 5 
HMI concepts Contro 722. 

Walking man 0 

Arrow 0 

Check 55 

LED strip 11.1 

Traffic 0 


Finally, the satisfaction was analysed thanks to the questionnaire after the tests. 
The questions to be answered concerned the subjective cognitive effort for the test 
participant to complete the task. This question had to be answered for each HMI 
concept as well as for the control group. In order to get a homogenous outcome, the 
test participants were asked to indicate the effort on a scale which ranges from —60 
(very low effort) to 60 (very high effort). Consequently, negative results indicate a 
low effort to complete the task. The answers revealed that only the control group’s 
value was distinguishable from HMI concepts, whereas the values of the HMI 
concepts among one another did not lead to significant differences (Table 4). 

Further questions were insightful regarding the perception if the test participant 
was able to detect the HMI concept (Detection), if the HMI concepts influenced the 
decision making for crossing the street or not (Influence on crossing), and if the 
HMI concepts were comprehensible for the test participants. “Yes or No” questions 
were asked to evaluate aforementioned perceptions. Figure 5 presents the answers 
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Table 4 Subjective effort to cross the road for the HMI concepts and control group 


HMI concept Indicator of effort 
Control group —22.8 
Walking man —42.5 
Arrow —46.8 
Check —35.4 
LED strip —43.0 
Traffic —42.9 


Percentage of affirmative 7 
Content Evaluation 


100% 
90% + 
80% + 
70% + 
60% + 
50% 4 
40% + 
30% + 
20% + 
10% » 
0% + 


aWalking Man 
BArrow 
mCheck 
OLight Strip 
OTraffic 


Detection Influence on crossing Comprehensibility 


Fig. 5 Results of questions regarding detection, influence on crossing and comprehensibility of 
HMI concepts 


for the questions regarding the possibility to detect the HMI concepts, if the HMI 
concepts influenced the decision making on crossing the street, and if the HMI 
concept was understandable for the test participant. 

The bars represent the percentage of participants that answered the questions 
affirmatively. This means for instance that 100% of test participants could detect the 
HMI concept “Arrow” and 50% of test participant stated that the HMI concept 
“LED Strip” influenced their decision making on crossing the street. As it is visible 
in Fig. 5, the “Arrow” HMI concept had the highest rates in all three categories. On 
the other hand, the “LED Strip”, “Check”, and “Traffic” HMI concepts were the 
concepts that influenced least the decision making to cross the street. Furthermore 
the “LED Strip” and “Traffic” HMI concept were the least comprehensible concepts 
in the test participants’ subjective sense. This qualitative data indicates a higher user 
satisfaction when interacting with the “Arrow” HMI concept than with the other 
concepts. 
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5 Discussion 


The discussion merges the facts that have been brought to light by the case study 
regarding the HMI and the more general points about the Usability Tests within VR 
that have been found out during the development process of the methodology. 

Within the presented study, it was possible to demonstrate the need for an 
explicit HMI for the communication between AVs and pedestrians in ambiguous 
situations. Indeed, even though the HMI concepts did not have significant differ- 
ences in terms of efficiency, effectiveness, and satisfaction among one another, the 
results had significant differences compared to the control group without any 
explicit HMI concept. This could be highlighted within VR since it was possible to 
test the explicit HMI concepts in an isolated way without interfering factors like 
deceleration. In other studies, like the one from Clamann, Aubert, and Cummings 
(2017), who conducted research on explicit HMI for manually driven cars that were 
disguised as AVs, deceleration and gap distance were identified as the main indi- 
cators for pedestrians to assess traffic situations. We suppose that these indicators 
interfered with the accurate evaluation of explicit HMI concepts since they con- 
stituted unneglectable variables. However, the neglect of these variables was pos- 
sible within VR and thus the presented method is validated as suitable tool for 
evaluating communication concepts without interferences. 

Furthermore, Pillai (2017) conducted a study of implicit communication (i.e. 
deceleration behaviour) between AVs and pedestrians at zebra crossings within VR. 
Beyond the results that a “human-like” driving behaviour from AV’s side improves 
the interaction with pedestrians, the study revealed that explicit HMI concepts 
would have been useful for the test participants to further assess the situation 
correctly. This underlines the necessity for explicit HMI concepts. 

In further studies, it is planned to investigate a combination of explicit and 
implicit communication for better usability for pedestrians. Supplementary devel- 
opment can also accommodate further scenarios in which for instance more than 
one pedestrian are willing to cross the road, or pedestrians come from different 
directions. 

Regarding the methodology used within VR, the presented study has validated 
that VR is a suitable tool to conduct Usability Tests for the case study of explicit 
communication between AVs and pedestrians in ambiguous situations. There are 
drawbacks in using VR like limited immersion and absence of haptic feedback. 
However, VR enables high flexibility to create the environment and the scenarios 
and the tests could be conducted with ensured safety for the test participants. This is 
aligned with findings from Deb et al. (2017) and Pillai (2017). 

As a next step, a benchmark test is planned to be conducted to prove the validity 
of collected data within VR. In order to do so, a scenario will be created in real-life 
conditions as well as in VR and afterwards compared regarding congruence of 
results. 


A Tool, not a Toy: Using Virtual Reality to Evaluate ... 215 


Finally, it will be investigated if the presented methodology can be used for other 
case studies like for instance how information, related to AV technology (e.g. intention 
or detection) can prevent anxiety from passengers’ side inside an AV for public 
transport. 


6 Conclusion 


The present work highlighted the suitability of Virtual Reality as a tool to test 
usability, particularly in the context of communication between AVs and pedes- 
trians. On the one hand, when no functional AV prototype is available, the method 
worked as a quick, preliminary validation method for the presented case study. On 
the other hand, the case study showed that the absence of communication between 
driver and pedestrian needs to be compensated towards an explicit HMI. Even 
though there are no significant results about differences of efficiency, effectiveness, 
and satisfaction attributes among the tested concepts, the VR Usability Tests helped 
to evaluate and dismiss several alternatives. Regarding the case study, further tests 
are necessary to evaluate the influence of implicit communication for the decision 
making of pedestrians in the Av to pedestrian scenario. In order to further validate 
VR as a Suitable tool for the conduct of Usability Tests in general, additional case 
studies will be selected and additional usability attributes will be focused on. 
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Designing Spatial UI as a Solution Mm) 
of the Narrow FOV of Microsoft crest 
HoloLens: Prototype of Virtual 

Museum Guide 


Ramy Hammady and Minhua Ma 


Abstract Since the Augmented Reality (AR) headset ‘Microsoft HoloLens’ 
released in 2016, the academic and the industrial community witnessed an obvious 
transformation and changes in the perception of AR applications. Despite this 
breakthrough, most of the HoloLens users have explicitly reported the narrow field 
of view (FOV) that crops the virtual augmentation from the viewer’s sight to a 
small window of 34° (Bimber & Bruns in PhoneGuide: Adaptive image classifi- 
cation for mobile museum guidance, 2011). This limitation can result in losing 
pre-made functions and visuals in the AR application. Therefore, this study intro- 
duced attempts to design a spatial UI representing a way around the narrow FOV 
that HoloLens suffers from. The UI was a crucial part of AR museum system which 
was evaluated by 9 experts in HCI, visual communication and museum engaging 
studies. Results showed a positive feedback on the usability of the system and 
users’ experience. This method can help HoloLens developers to extend their 
applications’ functionalities with avoiding missing content. 


Keywords Microsoft HoloLens - Field of view - User experience - Usability - 
Mixed reality - Spatial UI 


1 Introduction 


Augmented Reality Head Mounted Displays (HMD) and smart glasses are repre- 
senting a revolution after the use of mobile devices. They started progressing to 
overcome the limited screens that users can see the blended worlds from. However, 
the limitation of the hardware and the image processing of the existed AR HMDs 
reflected a lack of usability and caused cumbersome to users (Hsieh, Jylha, Orso, 
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Gamberini, & Jacucci, 2016). Therefore, the pioneering of Microsoft HoloLens was 
depicted on providing the opportunity to explore environments with hands-free 
HMD’s capability (Evans, Miller, Pena, MacAllister, & Winer, 2017). It means the 
device has the mobility feature with being wired with an external device or another 
controller. Moreover, it provides a content registration by spatial mapping 
(Coppens, 2017). These distinguished functionalities offer a wide usage of potential 
application, which subsequently reflects on more exploration on the concept of User 
Experience (UX) on HoloLens applications. 

The spatial 3D user interfaces are commonly used in virtual reality applications 
(LaViola, 2008). However, in the non-see-through HMDs, users cannot see the 
physical surroundings (Bowman, Kruijff, LaViola, & Poupyrev, 2004). The privi- 
lege of using spatial interface is it not constrains by the typical 2D desktop on 
mobile screens. Moreover, it open prospects to interact freely in the open space 
(Billinghurst, Poupyrev, Kato, & May, 2000). According to the spatial mapping 
feature of HoloLens, AR is perfectly suitable for creating User Interface (UI) for 
location-aware applications (Hdéllerer, Feiner, Terauchi, Rashid, & Hallaway, 
1999). Thus, the UI could be allocated as a virtual layer in between the user and the 
physical world in the desired location. HMDs such as Microsoft HoloLens can open 
prospects of innovative spatial UI with collaborative interactions. The privilege of 
the collaborative interactions is to involve group of users to interact on the same 
share UI which reflects positively on content engagement and rich shared 
experiences. 

Interaction with the spatial UI in the see-through HMDs requires a careful 
consideration to some factors. Some of them relevant to the user performance such 
as distance to be travelled to interact, size of the potential manipulated object and 
the length of the user (Bowman et al., 2004). Also, other environmental consid- 
erations are needed to take in account such as the of existence of the obstacles, level 
of activities or movement in the physical environment. User characteristics also 
should be considered such as the cognitive measures—e.g. the ability to do inter- 
actions spatially and the physical considerations such as arm length. Finally, the 
system considerations—e.g. the frame rate in the real-time (Bowman & Hodges, 
1999). 

In spite of the uniqueness of holographic lenses of Microsoft HoloLens, the field 
of view is quite narrow as Bimber and Bruns stated it is 34°. However, Keighrey, 
Flynn, Murray, and Murray (2017) stated the lens have an FOV of 30° by 17.5°. 
This limitation of FOV was not satisfactory of HoloLens developers and users 
(Bright, 2015) as it makes the AR experiences not so immersive. Others claimed it 
is not suitable for the user’s peripheral vision (Hockett & Ingleby, 2016). Also this 
constrained FOV conflicts with Milgram’s definition about the adequacy of 
see-through displays should be achieved in its FOV (Milgram, Takemura, Utsumi, 
& Kishino, 1994). 
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1.1 Related Work 


Not many developers have focussed on the development of HoloLens applications 
yet as it is an emerging piece of technology and due to the expensiveness of its 
price. However, there are some novel applications that exploited HoloLens in 
different sectors. For instance, ‘HoloMuse’ that engage users with archaeological 
artefacts through gesture-based interactions (Pollalis, Fahnbulleh, Tynes, & Shaer, 
2017). Another research utilised HoloLens to provide in situ assistant for users 
(Blattgerste, Strenge, Renner, Pfeiffer, & Essig, 2017). HoloLens also used to 
provide magnification for low vision users by complementary finger-worn camera 
alongside with the HMD (Stearns, DeSouza, Yin, Findlater, & Froehlich, 2017). 
Even in the medical applications, HoloLens contributed in 3D visualisation pur- 
poses using AR techniques (Syed, Zakaria, & Lozanoff, 2017) and provide opti- 
mised measurements in medical surgeries(Pratt et al., 2018, Adabi et al., 2017). 
Application of HoloLens extended to visualise prototype designs (DeLaOsa, 2017) 
and showed its potential in gaming industry (Volpe, 2015, Alvarez, 2015) and 
engaging cultural visitors with gaming activities (Raptis, Fidas, & Avouris, 2017). 

The literature around UI in optical-see-through HMDs was not discussed fre- 
quently. Furthermore, the limitation of the FOV of HoloLens causes challenges in 
developing the UI and UX for “hBIM’ project, which leads to a rapid disappearance 
of the content from the user frustum view (Fonnet, Alves, Sousa, Guevara, & 
Magalhaes, 2017). Also in ‘Holo3DGIS’, the authors stated that their system cannot 
show the content in the user’s visual space (Wang, Wu, Chen, & Chen, 2018). Due 
to the lack of investigating this problem which is a major concern for HoloLens 
developers, we were motivated to find a way around to enhance the usability and 
the user experience. 

According to the later literature, this paper we introduce: 


1 A method to redesign and restructure a spatial user interface that can find a way 
around the limitation of the FOV. 

2 Exploring the interactions of the designed UI method towards the four cate- 
gories of outside factors that determined by Bowman’s factors (Bowman & 
Hodges, 1999) which are user performance, environmental consideration, user 
characteristics and system consideration. This exploration will be measure by 
experimental evaluation. 


Keeping in mind the main aim is to increase the usefulness and immersiveness of 
the AR application overcoming the hardware limitation. 
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2 System Overview 


In order to manifest our method, it is prior to build a practical prototype and this 
prototype was constructed for cultural heritage guidance. The application is meant 
to be developed is a simple, interactive and informative system to guide visitors in 
museums. This system required from the user to wear Microsoft HoloLens and to 
explore the virtual content through spatial user interface. These interactions should 
be occurred with the existence of an exhibited antique in order to superimposed the 
content of the system as a complementary guidance. 


2.1 Functionality 


Our goal is construct the most usable UI that is capable to achieve the simplest 
interactions needed. The user has to interact with a spatialized 3D models, text, 
images, videos and buttons. The user has to deal with: 


1 Doing air-tap or hand gesture on the floating virtual objects such as the virtual 
replica of the authentic. The user can explore the virtual item in 360° so they can 
rotate it by dragging to right and left to spit it. The antique should be remains on 
its position. 

2 Interacting with buttons of documentary images and scripts when needed by the 
visitor. 

3 Interacting with a virtual guide represent a character from the same context 
explaining the exhibited item as a real-time virtual narrator. Also the user has the 
ability to click on a set of buttons to play/pause/replay to control the narrations’ 
flow. 

4 Interacting with small circles working as trigger buttons to reveal information in 
particular spots in the virtual replica. User can click on it by moving his head 
straightforward by the gaze point then doing the air-tap. 


2.2 System Architecture 


The system developments always be constructed by creating 2D and 3D assets. 
Development of the system took three stages as depicted in Fig. 1: 


1. 2D and 3D assets creation: In order to design a tempting and futuristic spatial UI 
design, 2D software adopted such as adobe Photoshop and Illustrator. All 
graphical items were designed and exported as PNG and JPG format which is 
going to be imported later as texture assets. Then, the 3D assets come to the 
process by creating the 3D model of the virtual guide using zBrush and 
Autodesk Maya. After modelling, the 3D designs required high textured 


Designing Spatial UI as a Solution of the Narrow ... 221 


2D Assets 3D Assets Creation 
oe. 

BEI 

UI 2D Design 

i SEEELL ES 


Museum Guide Development 
=> Create Gaze Point 


ei m> <4) unity => Create Spatial Mapping 


m> Oevelop Buttons Functionality 


30 Textures Guiding Sound : A : 
Models Graphical Narrations Effects -: ST Escan souwna 
Elements 8 
Compiling and Deploying 


Visual Connect to 
Studio —_ Holotens 


Compiling 


Find Location 
Spatial Mapping 
Allocating the UI 


Perform 
Interactions 


Fig. 1 System architecture 


materials so Substance Painter was adopted to accomplish this mission. 
Acquiring the virtual replica of the authentic piece went through using 3D 
scanners such as 3D sense and the mobile application ‘123D Catch’. Then, 3D 
scanned replica files refined and re-edited by Recap 360°. All 3D files are 
exported as FBX in order to be imported to Unity3D. 

2. Development Phase: All FBX will be imported to Unity3D project accompanied 
with PNG and JPG files that is considered as the main elements of the UI design. 
Add to it all; narrations audio files and sound effects libraries. In unity3D, 
project is created and the UI arranged in the scene and a set of steps are required 
to develop the system. Firstly, creating the gaze point in order to help the user to 
aim to the button that needed to be triggered. It usually appears straight ahead at 
the centre of the user’s vision box. Secondly, the spatial mapping is developed 
to scan the actual location in order to reallocate the designed UI and the virtual 
guide location next to the actual antique. It is designed to respond to the user’s 
hand gesture wherever interact. Thirdly, adding functionality to the spatial 
buttons such as images, scripts and controlling the narration flow. Fourthly, 
applying the navigation and orientation of the virtual replica in order to be 
controlled and responded by the user. 
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3. Compiling and Deploying: After building the application from Unity3D, Visual 
Studio’s role is to compile it and deploy it to Microsoft HoloLens through a 
wired connection to it. Then the headset will be ready for being used. Once the 
user opens the application, he/she can perform the interaction — hand gesture- to 
allocate the spatial visuals in the desired location and the virtual guide starts at 
once. 


2.3. UI Design Process 


The first idea to design the spatial UI design is to be designed as a half curve and all 
visuals are surrounding around the user. This approach was adopted in order to 
make all interactive points more reachable to the user and to ease the way of 
interactions. As depicted in Fig. 2, the brightened area is what actually the user can 
see from the whole scene and the semi-blacked area represents the unseen parts of 
the scene. In reality the blacked area is representing the actual environment without 
virtual content but the figure is to manifest the problem of missing content due to 
the narrow FOV. 

Unsurprisingly, and due to the limited FOV, what was seen from the HoloLens 
view port was clipped scenes as depicted in Fig. 3. Because of this uncoherent UI 
scene, users can not notice the existence of the missing content whether it is in the 
left or the right side of the user. Therefore, a series of experimental methods are 
conducted based on Bowman and Hodges (1999) outside factors which was part of 
their study. 

However, there are some other factors were found out aside with previous 
factors to accomplish the targeted goal. By considering all these factors that inte- 
grated with testing interventions, an obvious learning curve was noticed and we 
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Fig. 2. Spatial UI design as seen from HoloLens 
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found a solution to enhance the usability and interactivity. Depicted in Fig. 5, the 
UX principles diagram required for HoloLens UI design. 


1. Task characteristics: as described by Bowman & Hodges (1999), they represent 
all aspects that influence the performance. In our UI prototype, the user has 
several activities that affect on the way of performing towards it. User has to 
walk to the UI, aim with his/her head and centre the gaze point then do air-taps. 
In addition to look around and watch people walking while observing the 
authentic exhibited item. Moreover, the interactions with the virtual replica 
requires a dragged hand-gesture to rotate the object. These set of activities 
require specific attributes to be taken care of: 
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7 UX Principles to design for HoloLens UI 
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Fig. 5 UX principles for HoloLens UI Design developed from (Bowman & Hodges, 1999) 


(a) 


(b) 


Distance to be travelled: due to the limited FOV, what the user can see is 
merely quarter of the scene as mentioned before. Moreover, the user has to 
see the exhibited antique with the UI together. Also, it was calculated the 
best distance has to be from the UI is | m in order to perform a proper hand 
interaction with the UI. However, in order to see the entire desired scene, the 
user has to move back 2.2 m to see it as depicted in Fig. 4. The challenge 
was in the scene triggers that supposed to be so close to the exhibited 
antique. At the first attempt, we allocated the scene triggers 1 m away from 
the exhibited item. Unsurprisingly, this attempt results unrecognising the 
entire scene by the users however, we added a voice command or instruc- 
tions to look left and right. The second attempt went better than before as we 
allocated the scene triggers 2.2 m away from the exhibited item. However, 
the UI was so far and the location of the triggers were not in the desired 
location but users could see the whole scene. With minor voice and visual 
instructions, users came closer to interact with the UI and could realize the 
existence of the whole scene. 

Size of the object being manipulated: Based on our measurements in the 
intervened test of our participants, the best distance to do the interaction is 
1 m. Moreover, the most appropriate size that our participants felt conve- 
nient with is to be over 50 cm height and 50 cm width. It is worthy to 
mention that most of the participants were exposed to minimal instructions 
on how to do the air-tap and make the dragging gesture. 


2. Environmental characteristics: The environment of using and interacting with 


HoloLens requires an adequate space. Therefore, we consider the characteristic 
of the space by involving some variables in the process of interactions. 
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(a) 


(b) 


Number of obstacles: While running the application, it was concluded that 
the user requires an empty area in front of him/her to place the visuals. If 
some people pass in front of the user, it might change the location of the UI 
and that’s due to the deforming of the spatial mapping of the actual location. 
So, this problem is commonly in museums, it is expected to have many 
visitors at the same spot. 

Lighting levels: It is preferable to display visuals in low levels of lighting 
conditions and the visuals opacity increases with interior lighting condi- 
tions. Sun lights decreases the opacity percentages and the visuals start to 
lose its opaqueness. 


3. User characteristics: All aspects related to the user himself/herself regarding the 


physical and cognitive attributes. 


(a) 


(b) 


Cognitive measures: Participated group were instructed minimally the way 
the interact and do the air-tap. During the experimental phases, we noticed 
different levels of acquiring the interaction skills, which reflects an uneven 
retention of the instructions. Getting used of the HoloLens interactions takes 
time with some people and no time with others. 

Physical aspects: The diversity of the participants’ heights were noticed 
during testing interventions. We designed the visuals are appropriate to 
1.70 m for the person’s height. However, we noticed that shorter participants 
tend to look up to the visuals, which is a cumbersome for them and cause pain 
in their necks after time. Likewise, the participant who their height is longer 
than 1.70 cm. After several attempts, we were driven to scale the whole UI 
based on the person’s height. Once the scene opened, it calculates the distance 
between the ground and the camera of the HoloLens. Then, it scales the whole 
UI based on it. Eventually, it made the participants feel more convenient. 


4. System characteristics: All aspects that relevant to the headset or the application 


developed or the hardware specifications. 


(a) 


(b) 


Frame rate of the scene: It was noticed a rapid streaming of the physical 
visualisation that combined with the virtual content when the complexity of 
3D graphics is in minor levels. On contrary, if the current frame that the user is 
observing from the HoloLens viewport has loads of complex 3D models, the 
frame rate will come to 15—20 frame per second. It also could cause lags and 
delay of rendering the current frame. So, from the spatial design perspective, it 
is recommended to distribute the complex 3D visuals around the space with 
adequate room in order to avoid seeing them together in one frame. 

Visual and auditory instructions: From the UX concepts, the user should be 
aware of the all visuals that designed to be seen or to be heard. To loose 
recognising or seeing a visual content from the user cause a lack of user 
experience. Therefore, we used a ‘tag along method’, which gives the user a 
visual clue that points to the location of the virtual content in the space 
around the object. This method adopted by Fonnet et al. (2017). It ensures 
that the content is constantly a glace away from the users. We also used 
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auditory instructions to compensate the visual instructions if there is no 
room for the later method. 


3 Prototype Evaluation 


We opted for a field study demanding the involvement and participation of actual 
users. The aim of the evaluation is to investigate the usability aspect and the other 
unexplored aspects that might occur during the system usage. The evaluation’s 
nature was a completed simulation of the actual system that could be applied 
practically in museums. 


3.1 Method 


We conducted an evaluation in the university library where we invited 9 partici- 
pants and they are experts in different disciplines such as in Human Computer 
Interactions (HCI), visual communication and museum engaging studies. The 
research employed experts to do a discrete evaluation on aspects some of them were 
common to the participant evaluation and other relevant to their expertise. Worthy 
to mention that some similar studies employed disciplines’ experts to ensure the 
validity to the evaluation process (Karoulis, Sylaiou, & White, 2006). They were 
instructed prior the experiment by a brief tutorial on the way of interaction as 
depicted in Fig. 6. The evaluation method adopted the quantitative method as the 
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Fig. 6 Participants testing the system 
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Table 1 Participants demographics 


Discipline expertise Years of 
experience 
Academic and professional expert in Visual communication | F 22 45-60 
and Arts 
Expert at public engagement in museums F 7 31-45 
Expert in museum curatorship M 7 25-30 
Expert in museum curatorship F 6 31-45 
Expert in museum curatorship F 4 25-30 
Expert in HCI and visual interactions F 9 31-45 
Data manager and responsible for enhancing the museum | F 2 31-45 
visitor engagement 
Expert in museum curatorship M 10 31-45 
Academic and professional expert in museum curatorship | M 8 31-45 


research instrument was semi-structured questionnaires that include open questions. 
These questionnaires were piloted prior the experiment through other experts to 
make sure the questions are clear and answer the research objectives The quanti- 
tative method comprised of two types: textual analysis and numeric analysis. The 
textual questions were designed based on 1-5 Likert scale. The experiment took 
from 5-10 min per participant. In Table 1, the demographics of participant is 
manifested based on their age group, expertise area and years of experience. 


3.2 Results 


The numeric data in Table 2 represents the measurements of the usability aspect 
including the experience of interactions and preforming the functions desired. The 
first question was the lowest positive percentage of respondents were about the 
convenience of the headset and if it is comfortable during the usage of the prototype 
or not. As participants stated “Slightly heavy perhaps” and another said “Jt was a 
bit heavy on my neck, so I would not want it on too much”. It is worthy to mention 
that the headset is 579 g (Microsoft, 2015). The following question wonders if it 
causes dizziness, or headache, however the responds were highly positive. 
A participant commented “The HoloLens was much better than VR headsets, there 
was no disorientation or loss of the horizon, I was immersed in the location without 
losing track of my surroundings. Good experience”. The 3rd question’s responses 
also were quite positive, however a participant “J was glad to be reminded to look 
up and down”. The 4th question was regarding the air-tap and the comments were 
quite convincing. A participant stated “T had to learn how to do this—but that was 
part of the fun” and another commented “Yes, after minimal guidance”. The i 
question was examining the core of our theoretical study and the comments were 
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Table 2 Usability Aspect 


Measure Strongly | Agree | Neutral | Disagree | Strongly | Mean | Std. 
agree (%) (%) (%) disagree dev. 
(%) (%) 

I found the headset 0.0 771.8 11.1 4.3 3.67 | 0.707 

comfortable to wear 

I did not experience | 77.8 22.2 0.0% 0.0 0.0% 4.78 | 0.441 

nausea, dizziness, or 

headache 

I could look around 10.6 44.7 38.3% 6.4 0.0% 4.44 | 1.014 

the room comfortably 

I could do air tap on | 33.3 55.6 11.1 0.0 0.0 4.22 | 0.667 

the virtual objects 

appropriately 

I could interact with | 44.4 33:3. 11.1 11.1 0.0 4.11 | 1.054 

the user interface as I 

expected 

I could do all 33.3 55.6 55.6 11.1 0.0 4.11 | 0.928 

functions I desired 


(1 = Strongly disagree to 5 = Strongly agree. N = 9) 


progressive as a participant commented “Much more interactive than anticipated- 
love that you can more around the scene and look in all directions”. However, 
another participant stated “Jt is required some time to deal with it”. The 6th 
question also examines the main theoretical contribution and the percentage was 
similar to the previous question. Participants responses regarding this question were 
all positive as someone stated “This became easier the move I used the device” and 
another commented “Generally, very easy to use”. 


4 Discussion and Conclusion 


This research contributes theoretically by forming the UX principals for 
HoloLens UI applications as way to overcome the UI problems that stated in the 
literature. Our experiment was driven due to the formed UX principles in, which 
can enhance the usability aspect which was affected by the narrow FOV of 
HoloLens that was stated clearly in the literature. The conducted experiment 
introduced an empirical evidence through a HoloLens system prototype, which 
provides strong support for our hypothesis. Hence, this prototype was developed 
according to the UX principles in order to measure the ability to overcome the 
FOV UI problem. 

Also this paper manifests the system structure with the pipeline of the designed 
prototype that can be used to guide visitors in museums. This system is featured to 
have a personalised virtual guide, intractable virtual replica and holographic UI 
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presented aside with the exhibited antique. According to the positive results, the 
prototype proved its usability and accessibility to a group of experts with different 
expertise that relevant to the prototype cross-disciplinary nature. The theoretical 
contribution is quite helping the UX designers and developers to overcome the 
mentioned problem if they can consider these principles during their pipeline and 
testing phases. Also the techniques are flexible to be applied to any application has 
the spatial UI design and intractable floating buttons. The UX model also consider 
users who has different heights and also consider different environment natures. 

Despite the HoloLens enabled us to develop prototypes with genuine AR 
applications, it still has limitations that should be considered in the future versions 
of it. For instance, the unstable spatial mapping function was witnessed when 
someone crossed in front of the user. Also the device is bulky for some users, so 
they could not able to wear it for long periods which means it is not so much 
acceptable in museum long tours. Moreover, the real-time rendered frame should 
not have so much graphics—3D models and textures-, otherwise it could cause lags 
and delays of real-time rendering. 

Enhancing the museum experience through immersive technologies reflects 
explicitly on the tourism industry. It absolutely encourages visitors to come to 
museums to experience the technological tool and to enjoy it. Hence, museum 
managements need to invest time and resources on these vital tools as it might be 
the next revolutionary gadget in museums that can reshape the mental image of 
museums. 


4.1 Future Work 


We seek for adding more usability and enriching the user experience to the system 
so that we are aiming to add voice recognition and text-to-speech techniques. We 
still not sure if the voice recognition can practically work with some languages e.g. 
Arabic language. Also we envision a futuristic AR system that can present visuals 
beyond the spatial UI and can depict holograms everywhere around the user. That 
system can consider the user is in the centre of a sphere of interactive worlds. We 
also aim to include games at specific zones of museums to enrich the level of 
interactivity and empower the content with contextual interesting stories. These 
games should have some motivational actions such as exploring and investigation 
the rooms to discover clues for rewards and reaching milestones. These actions can 
encourage the physical interactions within the museum context. Hopefully, this 
expanded system could achieve more aims with newly developed HoloLens or 
headset with widen FOV. 
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Recommender System as the Support ®) 
for Binaural Audio cee 


David Bernhauer and Tomas Skopal 


Abstract Virtual reality devices nowadays can effectively utilise other senses 
besides vision, too. The most often used secondary sense is hearing with binaural 
audio as VR engine. Currently, practical usage of binaural audio as the source of 
VR is impossible because of the inaccuracy of a general model. On the contrary, 
measuring the personalised parameters can be time-consuming. Our task was to 
prove the possibility of reconstruction of the binaural audio parameters in domestic 
conditions. We have focused on the design of the user interface that can be used 
independently on the platform. Our proposed browser-based application uses col- 
laborative filtering as a recommender system. We have proven that sound-based 
navigation in axial plane is possible with 6.6° inaccuracy. The gamification and 
browser-based implementation make it easier for all people to find the best possible 
parameters. The resulting profile can be used both with fully VR environment and 
with semi-VR games. 


Keywords Binaural audio - HRTF - Recommender system - 
Collaborative filtering - Virtual reality 


1 Introduction 


Virtual reality is currently widespread mostly with just the graphical part of 
applications. Developers must deal with the inaccuracy of audio component. 
Currently, there are two most used approaches. Firstly, they can completely ignore 
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spatial audio. This variant is suitable for application where the sound has a support 
role, such as background music or pre-recorded 3D sound. On the other hand, 
developers want to provide complete immersive experience. In most cases, it is not 
necessary to have an absolute precise simulation of spatial audio. Binaural audio 
(with right parameters) can provide the best results. The primary parameter of 
binaural audio is head-related transfer function (HRTF) which represents the 
hearing of the user. The usage of pseudo-generic HRTF profile achieves average 
results across all users. 

Some applications, including assistive technology and audio-based entertain- 
ment industry, need more precise spatial audio simulation. The audio part of virtual 
reality should provide high enough precision for navigation in space to substitute 
for vision. The binaural audio can be used in the same manner. However, HRTF 
needs to be personalised for every subject. The precise measurement of the per- 
sonalised HRTF can be time-consuming and uncomfortable. 

There are various HRTF databases without the wider practical use. Their primary 
purpose is to support alternative ways of HRTF reconstruction based on different 
measurements. Anthropometry is a method based on the relationship between ears 
and head measurements and HRTF. 

It is obvious that some profiles match the user better than other profiles. The 
simple idea is to find the best HRTF for the user instead of using a generic one. 
Recommender systems are widely used in business, mostly by e-shops. Specialised 
recommender systems can suggest results based on content, user ratings or more 
complicated formulas (including HRTFs). 


1.1 Contribution 


The main purpose of the work is to analyse possible methods of ranking HRTFs 
and prove that collaborative filtering can efficiently choose the best matching 
HRTF. Collaborative filtering can later be switched to another recommender 
system. 

We have looked for a solution that will provide nearly-precise simulation, save 
the time of the user and be comfortable. We have focused on application that will be 
easy to use in domestic conditions. It has been our priority to minimise special and 
expensive device requirements. 


2 Physics of Binaural Audio 


The principle of binaural audio is based on acoustics. We can model audio space 
and listener properties. The main idea is to divide the source of sound into small 
sources. The human ear cannot distinguish the original from the small one near the 
ear. In the same manner, the camera cannot distinguish the farther silhouette from 
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al 


Fig. 1 Camera (left) can be cheated by a smaller closer object. Binaural audio (right) can process 
the signal in the same manner 


the smaller but closer one as shown in Fig. 1. It means that we can replace the 
original source by the expected signal in ears. 

This chapter describes basic principles of spatial audio. Combination of these 
principles makes binaural audio more realistic. In the end, we can find an overview 
of current approaches, advantages and disadvantages. 


2.1 Basic Principles 


The common and most intuitive approach to modelling spatial audio is a simulation 
of different level of sound. It is obvious that the listener will perceive 
right-positioned sound louder from the right-side. This fact is called interaural level 
difference (ILD). Interaural time difference (ITD) compares delay between left and 
right ear. Both variables directly depend on the head size and shape. 

ILD and ITD can be represented as a function of relative position of the source 
(spherical coordinates), sound and head (shape). Detailed model of the head may 
not be available for practical use. Moreover, ILD and ITD provide precise sound 
localisation only in the axial plane. The cone of confusion shows equally distant 
points as shown in Fig. 2 (right side). 

Head-and-torso approach brings another point of view. Torso can provide 
additional information about the elevation of the sound position. The sound will be 
reflected from the shoulders as shown in Fig. 2 (left side). The ears will receive the 
same signal two times. The first signal will determine a set of points in the cone of 
confusion. The second one will determine the elevation of the source. 
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Fig. 2 Head-and-torso reflections with delay (left). Same distance (ITD, ILD) in the cone of 
confusion (right) 


The complexity of the human ear reflects the precision of determining the 
position of the sound source. Unlike bats, humans cannot map space by echolo- 
cation. Even so, they can navigate in space with the support of a navigator. Human 
ears as the complex device utilise the high number of pinna reflections for sound 
localisation. These ear details are almost unique for everyone (similar to 
fingerprint). 

Combination of mentioned principles can be generalised as a function of relative 
position, person and signal. All of these reflections and modifications encode the 
position of the original sound in head-related impulse response (HRIR). The 
head-related transfer function (HRTF) is a function that characterises how the final 
signal is modified. Generally, HRTF is Fourier transform of HRIR. 


2.2. Standard Approaches to Retrieving HRTF 


In practice, we can divide standard approaches into two categories. The first cat- 
egory is measuring HRTF. Measuring HRTF requires specialised software and 
hardware including a soundproof room. The basic principle is the same as men- 
tioned above. Microphones can be placed in the ear canal, and HRIR is recorded for 
every position. Alternative approaches try to deal with the time-consuming process 
of recording. Time efficiency can be achieved by swapping the microphone for 
speakers at the expense of discomfort. 

Because of unavailability (and other disadvantages) of public HRTF measuring, 
alternatives are needed. The second approach is modelling the HRTF based on the 
other properties. Current approaches are discussed in Chap. 3. Our method pro- 
poses the possibility of reusing existing HRTF databases for almost precise HRTF 
retrieval. 
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3 Related Work 


The most accurate method is measuring the HRTF for the particular person. The 
measuring is time-consuming and uncomfortable. The measuring itself has to deal 
with inaccuracy caused by noise and imperfections during measuring session. The 
problems during measuring are discussed in (Plaskota & Pruchnicki, 2014). 
Measuring is not always the solution to the problem. Especially, measuring HRTF 
near a subject’s head is a little bit problematic, because the loudspeakers used for 
producing white noise cannot be considered as the point source as written in 
(Hosoe, Nishino, Itou, & Takeda, 2005). 

Another popular method is the reconstruction of the subject’s HRTF based on the 
anthropometry. This approach expects that HRTF depends directly on the geometric 
model of the head and torso. In this work, we have used CIPIC, one of the open 
HRTF databases (Algazi, Duda, Thompson, & Avendano, 2001). Some theories 
work just with the pinna anthropometry. Zotkin et al. (2003) describe the differences 
between pinna and head-and-torso anthropometry and the consequences. 

Current approaches involve modern principles and are based on recommendation. 
We can find experiments based on autoencoder recommendation and modelling. Luo 
et al. (2013) used a neural network for recommendation and individualisation of 
HRTF. The problem of many similar experiments is that the real user is replaced by a 
virtual one. A virtual user does not have the same qualities as a real one. Yamamoto 
and Igarashi (2017) bring an experiment with autoencoders and real users. The users 
subjectively choose the better HRTF. In our work, we show another method of 
recommendation and testing. 


4 Recommendation System 


Today’s industrial approach involves usage of a general HRTF profile. 
General HRTF profile can be retrieved by aggregation as an average one. The basic 
idea is to utilise public HRTF databases. Recommendation of the best HRTF can 
depend on many factors. It is impossible to recommend precise HRTF because of 
its uniqueness. 

Proposed approach involves a simple recommendation system. Recommendation 
systems are based on finding relationships (similarity, dependency, complementar- 
ity). The recommendation can be used in many fields. Collaborative filtering spe- 
cialises in finding similarities based on ratings of users. 

The principle of collaborative filtering (CF) is finding similarities between 
persons and objects. If person P, likes object O;, and person P, and P3 are similar, 
then we can expect that P, will like O; as shown in Table 1. Similarly, we can 
define an inverse thesis about objects. The core of the problem is specification of a 
similarity function. 
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Table 1 Like-table for CF 


Py Like Dislike Like 27? 
P, (similar P;) 22? (exp. Like) Dislike Like Like 
P3 (dissimilar P,) 27? Like Dislike Dislike 


The question mark: the user does not rate the object 


The tricky part is the word “like”. The user has to rate his relationship to the 
object. There are many possibilities: yes or no, from 1 to 5, purchase ... But, it is 
sometimes impossible to rate the item. The user cannot decide if one HRTF is better 
than another one. The next section describes methods how the user can directly rate 
HRTF profiles. 


4.1 User-Related Rating Methods 


User’s ranking is essential as the primary input of the collaborative filtering. The 
aim of collaborative filtering is to find the best matching existing HRTF from a 
database. 


4.1.1 Laboratory Method 


The first discussed method focuses on the laboratory approach. The primary target 
is to provide maximum flexibility for the expert to control the whole testing session. 
The measured subjects are sitting in the laboratory with the standardised earphones 
(or headphones) in their ears. The expert randomly or systematically chooses the 
position of the sound source. The spatial sound is reproduced to subject. The 
subject will record the assumed position of the source in a simple application or on 
a sheet. The recommender system will regularly evaluate the error and improve the 
subject’s HRTF recommendation shown in Fig. 3. 

The advantage of this method is the fully controllable environment. The expert 
can modify any parameter of recommendation system at the time of testing and 
improve the recommendation. The whole process can be interrupted or restarted in 
case of any problem. The testing environment provides information about the 
precision of simulation for different positions. The successful laboratory method 
combined with measuring the HRTF can provide proof that a recommender system 
will find the optimal HRTF for the subject. This makes this approach suitable just 
for experimental purposes. 

The disadvantage of this method is in the way of recording the assumed position 
of the source. The subject is forced to identify the position by hearing, but he 
records it by sight. This method does not correspond with the localisation of sound 


Recommender System as the Support for Binaural Audio 239 


Initialize remaining test cases 


Prepare Adjust system 
test case parameters 
Evaluate 
Play soun ee 


experiment 


Virtual space (application) Reality (paper) 


Position Position 
(original) (recorded) 


Ay 


Listen sound 
seinen Record expected 


& 
Identify position Ee 


Fig. 3. The activity diagram shows laboratory approach and the problematic difference between 
virtual (audio) environment and a physical paper sheet 


in the real world. It does not allow him to move his head. Secondly, the problem is 
switching between sight and hearing like switching the lights on and off. The whole 
experiment can be uncomfortable for the subject. These problems make this method 
unusable for wide use. 


4.1.2 HRTF Reconstruction in Home Conditions 


This method is focused on reconstruction in typical home conditions. It assumes 
standard low-end computer or notebook, low-end headphones and internet con- 
nection (optional). The subject will start the application (alternatively open the web 
application), read the instructions and start the recommendation process shown in 
Fig. 4. 

The user should be able to install an application or open web page. He is placed 
in the virtual environment and somewhere is a hidden target. The user can rotate the 
virtual avatar’s head with a mouse. If he is sure that he is targeting the source, he 
will record the position by mouse click. This process is repeated until precision of 
HRTF is not suitable or time limit exceeded. 
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Fig. 4 The activity diagram shows domestic approach. Original sound and recording of the 
expected position are in the same application 
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The advantage of this method is wide use in home conditions. It does not require 
any special hardware requirements. The subject doesn’t need to convert audio space 
to visual space. The whole process is easily understandable to an average user. 
Adding advanced hardware can improve the recommendation process. 

The main problem is the assumption that the subject will follow the instructions. 
Secondly, there is no expert present. Depending on the implementation, the expert 
can be connected remotely. Remote control allows him to partially modify the 
parameters. 


4.1.3. VR-Extended Method 


The last discussed and analysed method involves usage of virtual reality device. VR 
device makes wide use more complicated. VR devices are expensive for most 
people today. On the other hand, it provides different approaches to recording the 
position of the sound. VR devices should be considered from the long-term view. 

Typically, we can use a head tracking device instead of a mouse. Head tracking 
makes the simulation more traditional. More natural behaviour means more precise 
HRTF recommendations. VR sets with hand motion tracking device should be more 
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interactive. With this kind of a tracking device, it is possible to test localisation in 
different positions by pointing the hand only. The subject will get complete VR 
experience by combining the hand and head motion tracking. 


5 Experiment 


The experiment of the efficient recommendation of the most personalised HRTF 
was designed for use in home conditions. The testing web application was 
implemented in WebGL and WebAudio. Minimum requirements are a low-end 
computer with a modern browser, internet connectivity and headphones. The pro- 
cess of the experiment is hidden behind the gamification. The player (subject) has to 
shoot the enemy (source of sound) and maximise the precision and number of hits 
in 15 min. Firebase was used as a database for the collected data. The Firebase 
database makes the experiment semi-supervised in real-time. 

As the recommender system, we have used collaborative filtering based on 
similarity of users. Because the number of users will grow and the number of 
HRTFs will be constant, it would be logical, from a long-term perspective, to base 
the collaborative filtering on a similarity of HRTFs (Lops, De Gemmis, & 
Semeraro, 2011). As the HRTF database, we have used CIPIC database, which 
provides additional data about the 43 original subjects. 


Users. 40 users participated in the experiment. 25 of them finished both phases of 
the experiment. Testers were mostly students of the Czech Technical University in 
Prague. About 12% of participants were women. 

Phases. The experiment was split into two phases. The purpose of the first phase 
was to get an average precision error with the usage of a pseudo-general HRTF 
profile. The pseudo-general HRTF profile was simulated by recommending uni- 
formly distributed random HRTFs. The second phase happened after two weeks. 
Almost every subject completed this phase at home. During the second phase, the 
HRTFs were recommended by collaborative filtering. 


5.1 Results 


During the first phase, the subject’s precision highly depended on the randomly 
chosen HRTF. The average inaccuracy was approximately 30°. The subjects 
acknowledged that they were confused most of the time and they were not able to 
accurately localise the source of the sound. The average number of hits in 15 min 
was around 50. Example accuracy during the first phase is shown in Fig. 5. 
During the second phase, the subject’s precision was at first inaccurate similarly 
to the first phase, but in the second part (at least last 20% of hits) of the experiment, 
the average inaccuracy decreased to 6.6°. Nearly all subjects improved over the first 
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Fig. 5 Subject #16Q6yE inaccuracy during the first phase without individualisation. It can be 
seen the accuracy is highly dependent on current HRTF profile 
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Fig. 6 Subject #16Q6yE inaccuracy during the second phase of collaborative filtering. First 15 
shots serve as the best solution searching. Final accuracy is stable 


phase, except the subjects that were near the 6.6° limit in the first phase. The 
average number of hits in 15 min was nearly same. Example accuracy during the 
second phase is shown in Fig. 6. The final results are presented in Table 2. 
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Table 2 Average inaccuracy in individual parts of the experiment 


Standard error 
4.097° 


Experiment phase Average inaccuracy 


5.653° 


Visible target (without crosshair) 


HRTF without individualisation 29.712° 44.256° 
HRTF with collaborative filtering 13.995° 17.256° 
HRTF with CF, last 20% hits only 6.632° 2.243° 


5.2. Problems 


In contrast with theoretically oriented works, we have focused on problems that 
cannot be included in probabilistic models. We can observe many different patterns 
of behaviour during the testing session. Our solution must take these into account. 
Based on analysis results we have implemented several features to avoid these 
problems. 

The biggest problem with the whole process is human error. As the collaborative 
filtering is highly dependent on the rating, a wrong rating can result in a wrong 
recommendation. The wrong recommendation leads to rejection of a potentially 
perspective solution. In our case, we discard the best and the worst rating as we 
expect that these outliers can be a statistical deviation. The misclick problem is 
shown in Fig. 7. 

We have predicted that the user will be confused in the front and back direction. 
This confusion may arise due to many reasons. The primary reason is that the user 
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Fig. 7 Subject #L6R7Ep inaccuracy caused by misclick. The peaks represent the misclicks which 
caused worse accuracy 
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Fig. 8 Subject #LfApOt inaccuracy caused by disorientation 


is not used to orienting themselves with hearing. The different way of localisation 
may confuse many users at the beginning. The second reason is not being used to 
the mouse control. Most users move their heads in an attempt to find the source of 
the sound. The front-back problem is shown in Fig. 8. We have implemented first 
five rounds with a visible target to make sure that the user understands the controls. 
The expert was present during the first phase of the experiment. He could notify the 
user in the case of total disorientation. 

Another pseudo-problem during the experiment was caused by several users 
with good sound perception. These users could adapt to almost every HRTF profile. 
In this case, it was very difficult to find the best HRTF profile and beat the accuracy 
of the first phase. Results from both phases were nearly the same. 

At last, one of the problems was the sound used for localisation. We have 
repeated one track again and again. The users were tired after few minutes. In 
practice, triviality like this can determine the success or failure of this method. 
Before expanding among the public, we should research further and try to find a 
generally pleasurable sound. 


6 Conclusion and Discussion 


Our experiment proved that HRTF recommendation is an efficient way of 
improving the accuracy of sound localisation. It is possible to get relatively accurate 
HRTF using a standard computer and modern browser. The average improvement 
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of our approach was 25°. The final accuracy was close to the limit of vision 
accuracy. The average number of hits was nearly same. 

The experiment required precomputed data from the first phase. In practical 
usage, we can provide it separately. Alternatively, we can extend the testing session 
to 30 min. First 15 min will use the random recommendation as the exploration 
of preferences. The other 15 min will be used for collaborative filtering 
recommendation. 

We have to deal with human error. Sometimes the subject may localise sound on 
the opposite side of space. This phenomenon is due to the fact that subjects are not 
used to hearing as a primary source for localisation. 

Collaborative filtering has proven to be an efficient approach to find the best 
available HRTF. From the long-term point of view, it is necessary to consider how 
to improve existing HRTF for better personalisation. This kind of extended per- 
sonalisation could be based on subject behaviour during the use of the recom- 
mended HRTF. 


6.1 Binaural Audio for Business Purposes 


Currently, there are no many applications using binaural audio. Existing applica- 
tions only use a general profile. It has just a support role. The biggest problem is 
that there is no easy way for users to record their HRTF. Precise HRTF recordings 
are used in assistive technologies. Most of these assistive tools are in experimental 
phase. 

Generator (or recommender system) of HRTFs can be the first step towards 
allowing users to use a personalised profile. This efficient approach can bring new 
type of application and game development industry. Augmented and virtual reality 
application can be extended by binaural audio and provide more realistic simula- 
tion. Game industry can easily extend existing games with spatial audio without any 
additional hardware requirements. 

Practical usage will need to define a standardised format for HRTF profiles. 
HRTF databases can rapidly grow and hold additional information (like anthro- 
pometry). The recommender system will improve as the requirements will raise. 
These changes can help the audio-based assistive tools too. 
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Virtual Reality References in Design Mm) 
Problem Solving: Towards ot 
an Understanding of Affect-Cognition 
Interaction in Conceptual Design 


R. Vimal Krishnan and Prasad S. Onkar 


Abstract Empirical studies in cognitive psychology indicate that a typical problem 
solving activity will involve both affective and cognitive processing. Design 
problem solving being a creative process is often driven by affective components 
such as moods, emotions and feelings. This paper demonstrates the dynamics of 
affective influence on mental imagery processing, a cognitive process involved in 
producing and modifying visual representations, in the context of conceptual 
design. This is done by foregrounding the impact of immersive technology such as 
virtual reality (VR) references through an experimental pilot study. Subjects who 
received VR references to solve a design problem, demonstrated a markedly 
different kind of affective engagement as compared to those who received 
screenshot references. This is reflected in the distinct styles of mental imagery 
processing and the consequent differences in the produced design solutions, which 
are presented in the paper as preliminary results. 


Keywords Affect - Cognition - Virtual reality - Conceptual design 


1 Introduction 


Human computer interaction (HCI) is in the wake of a paradigm shift brought about 
by emerging sociotechnical systems such as wearable devices, voice/haptic inter- 
faces and virtual/augmented/mixed reality setups. This shift has a far reaching 
impact on the psychological dimensions of users and designers who transact with 
such technical interfaces. Consequently, research in the field of design and HCI 
points to an increasing interest in the cognitive and affective sciences. Though a 
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considerable amount of work which links cognition, design and HCI has happened, 
the implications of affective processes is an emerging area of study. Jeon (2017) 
classifies the major research paradigms concerning affect, design and HCI into four 
broad categories: emotional design (Norman, 2004), Hedonomics (Hancock, Pepe, 
& Murphy 2005), Kansei engineering (Nagamachi, 1995, 2002) and affective 
computing (Picard, 1997). 

The role of affect is especially significant in research concerning virtual reality 
(VR) environments. VR systems are being employed in numerous domains and 
applications ranging from psychology, neuroscience, scientific visualization, edu- 
cation, sports, cultural heritage and so on. A fundamental aspect of VR which 
distinguishes it from other media is that it generates an emotionally engaging 
experience through illusions of “place (presence), plausibility and embodiment.” 
Due to the immersive nature of VR, participants perceive the simulated environ- 
ment as real and they react emotionally to stimuli contained within the virtual 
environment (Slater and Sanchez-Vives, 2016). Therefore a VR experience is 
inherently affective. Research focusing on the hedonistic, social and functional 
aspects of virtual reality and its implications for commercial sectors such as 
entertainment and tourism highlights how affective affordances of VR can yield 
economic benefits (Jung et al. 2018). 

The present study focuses on the influence of affective arousal triggered by 
VR references/mood boards among design students engaged in the conceptual 
stages of design problem solving. 


2 Review of Literature 


Design processes bring about the transformation of abstract ideas into concrete 
forms and solutions (Suh, 1990). A design activity typically involves designers in 
an iterative process between a design problem and a set of possible solutions (Dorst 
& Cross, 2001). The conceptual phase of design thus becomes very significant, as 
designers tend to develop numerous early ideas and solutions in this phase. The 
significance of various psychological processes activated in designers while they 
are engaged in the conceptual phase has been a topic of interest for many design 
researchers. 


2.1 Affect, Cognition and Conceptual Design 


Literature in the domain of psychology distinguishes cognition and affect as two 
separate modalities of mental information processing. While cognition denotes a 
more rational and logical approach to a problem, affect implies aspects such as 
emotions, feelings and moods. Affective and cognitive processes are deeply linked 
and influence each other (Zajonc, 1980; Lazarus, 1984; Damasio, 1994; Schwarz 


Virtual Reality References in Design Thinking Problem Solving ... 249 


and Clore, 2003). Though the influence of cognition on conceptual design has 
received much attention in design research (Goldschmidt, 1991; Goel, 1995; Suwa, 
Purcell, & Gero, 1998; Kavakli & Gero, 2001; Hay, Duffy, McTeague, Pidgeon, & 
Vuletic, 2017), only recently have researchers started to focus on the role of affect 
in the early stages of design ideation. Empirical studies in the domain of cognitive 
psychology document the multifaceted nature of affect-cognition interaction. But 
design research tends to treat cognitive processes in an isolated manner and studies 
related to the role of affect have largely been confined to the evaluative stages of 
conceptual design (Kim & Ryu, 2014; Hay et al. 2017). Suwa, Purcell and Gero 
(1998) in their macroscopic analysis of conceptual design activity, identified four 
major categories of actions that designers perform: physical, perceptual, functional 
and conceptual. Through their analysis, they revealed that designers engage in 
evaluative actions (e-actions) to judge aesthetic aspects of their conceptual sketches 
and ideations. Such aesthetic judgments are often driven by affective dispositions. 
There are other studies which show that emotional dimensions can be incorporated 
into new product designs by understanding emotional reactions of users to existing 
product designs. The affective user experience data may be used as starting points 
for an emotion driven design approach (Desmet & Dijkhuis, 2002). In a product 
design context, Leblebici-Basar and Altarriba (2013) demonstrate that incorporating 
“emotion” words into design briefs trigger more creative and varied associations. 
When designers tackle briefs which have emotional words and affective cues, they 
tend to produce more diverse translations of ideas into embodied forms. Kim and 
Ryu (2014) report on the relevance of affective heuristics which influence problem 
solving activities of expert designers. When tackling complex design problems 
expert designers often take design decisions based on like/dislike dispositions about 
artefacts and their own design solutions. Expert designers also rely on an “attitude 
heuristic” to classify existing design options into favourable or unfavourable cat- 
egories. A favourable or unfavourable attitude to a design option might determine 
its selection or avoidance which in turn influences subsequent design strategies. The 
importance of understanding the emotional impact of conceptual sketches made by 
designers and how such emotions can be transferred to the embodied form of the 
product has also been documented (Kim, Bouchard, & Ryu, 2012). Lindgaard and 
Wesselius (2017) also point out new trajectories of design thinking in the context of 
feeling and embodied cognition. They situate theories of design cognition in the 
framework of material engagement and indicate that physical dimensions of design 
such as sketching and the associated feelings triggered by such embodied actions 
need to be taken into account while theorizing design cognition. 

This study attempts to understand the dynamics of affective influence on cog- 
nition in a designer oriented manner, especially in the context of employing virtual 
reality references in conceptual design. 
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2.2 Affect, Virtual Reality and Conceptual Design 


This section discusses research pertaining to affect, virtual reality and conceptual 
design. Tenneti and Duffy (2006) establish the correlation of various rendering 
styles, (photorealistic, non-photorealistic and vague rendering styles) with user 
perceptions and affective responses. They show that specific rendering styles of 
computer graphics can be employed to convey different kinds of information such 
as affective, functional and cognitive in order to elicit the appropriate responses 
from viewers. Model based VR environments could therefore be rendered in a 
specific style, depending on the kind of affective responses to be elicited from 
viewers. 

The positive implications of utilizing an immersive workflow in an industrial 
design context have also been documented (Rieuf, Bouchard, Meyrueis, & 
Omhover, 2017; Rieuf, Bouchard, & Aoussat, 2015). The results of these studies 
show that using immersive three dimensional sketching environments and vir- 
tual reality mood boards can heighten the emotional engagement of designers. This 
in turn leads to better conceptual designs as designers transfer ideas from “inspi- 
ration” to “generation” in a typical design process. 


2.3 Research Questions 


As mentioned earlier, a typical design activity entails designers traversing back and 
forth between the problem space and the solution space, modifying both the entities 
in the process. Several factors influence this recursive movement such as the nature 
of the design problem/design brief (Rao, Onkar & Mathew, 2017), types of 
available visual information (Laing & Masoodian 2015, 2016; Laing, Apperley & 
Masoodian 2017) cultural background, experience levels of designers and their 
design settings (Lotz & Sharp, 2017), design tools (Rahimian & Ibrahim, 2011; 
Baskinger, 2008) and so on. Psychological processes of designers also play a huge 
role in influencing the manner in which they tackle a conceptual design task. 
Though there have been studies highlighting the various strategies adopted by 
designers when they solve a problem in the conceptual phase (Kruger & Cross, 
2006), the symbiotic interaction of affective and cognitive processes in design 
problem solving could also be studied. 

Clore, Schiller and Shaked (2017) have propounded three principles of 
affect-cognition interaction: the attribution principle (Affective responses are attri- 
butable to specific objects such as cognitive thoughts or external stimuli), the 
immediacy principle (affect is generated by immediately accessible mental phe- 
nomena and affective appraisals of such mental phenomena influence problem 
solving) and affective processing principle (positive or negative affect are analogous 
to decision boxes determining “YES/NO” values for cognitive inclinations and 
thought trajectories.) 
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The present study extrapolates such theoretical models of affect-cognition 
interaction to the context of conceptual design. The following hypothesis therefore 
stems from the need to discuss the role of affect-cognition interaction in a designer 
oriented manner in the conceptual design stage. 


RQI1: Do positivelyMegatively valenced affective states and high/low arousal levels 
influence mental imagery processing of design students engaged in conceptual 
stages of design problem solving? 


In addition to the foregoing hypothesis, this study also investigates whether 
specific tools such as virtual reality references play any significant role in altering 
the nature of affective engagement among design students. Existing studies on 
inducement of emotions and memory formations through virtual reality 
(Genheimer, 2015; Shin, 2018; Sch6ne, Wessels, & Gruber, 2017) indicate that 
exposure to VR simulations does result in affective engagement among users and 
clinical patients. This study attempts to extend these findings to a sample of design 
students engaged in a conceptual design task. 


RQ2: Do VR references/Mood boards influence the affective engagement of design 
students working on conceptual design ideas? 


3 Methodology 


3.1 Experimental Pilot Study and Expected Findings 


In order to capture the interaction of affect and mental imagery processing of the 
respondents, a sample problem statement was developed. In addition to this, ref- 
erence material in the form of screenshots (non VR) and 360° immersive video to 
be viewed in a mobile based VR device were selected. The respondents, divided as 
experimental (with VR reference) and control subjects (with non VR reference) 
were exposed to the reference material before they performed the design task 
outlined in the problem statement. The pilot study focussed on capturing the 
affective influence on mental imagery processing in two stages: (a) post exposure to 
the reference material (VR/non VR) and (b) persistence and influence of affect 
during problem solving/ conceptual sketching. This enabled the correlation of 
affective dynamics to the type of reference material. The impact of affect on mental 
imagery processing during problem solving could also be analysed. Mental imagery 
processing is a cognitive process which involves the generation, maintenance and 
transformation of visual mental images (Kosslyn, 1995). Mental imagery process- 
ing is analogous to visual perception in that visual representations are internally 
generated and maintained for further examination. But mental imagery processing 
is driven by memory and not perceptual sensory inputs. The subsequent impact of 
the interaction of affect and mental imagery processing on the nature of the pro- 
duced conceptual designs were also evaluated. To summarise, the study focuses on: 
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e Measuring positive/negative affective dispositions owing to the type of 
references/mood boards (VR/non VR). 

e Extending theoretical underpinnings of affect-cognition interaction available in 
cognitive psychology to design contexts. 

e Comparing the interaction of affect and mental imagery processing triggered by 
VR and non VR conditions with regard to differences in the produced con- 
ceptual ideas. 


3.2 Procedure 


A pilot study was conducted with an experimental research design, wherein four 
master level design students (2 male and 2 female students) with varied back- 
grounds participated. The experimental subjects received a design brief and a VR 
reference material to generate conceptual sketches for a graphic design task. The 
control subjects received the same design brief given to the experimental subjects. 
But instead of a VR reference material, screenshots gathered from the VR footage 
were given (Table 1). 

The two pairs of respondents had to execute the given design task in a time 
frame of 30 minutes (excluding time taken to read the brief and exposure time to 
reference material) by producing conceptual sketches using a digital tablet 
(WACOM INTUOS) and design software of their choice (Adobe Photoshop/ 
Illustrator CC Trial Versions). The participants in both the conditions were given 
the design brief (to be read in 5 minutes). They were then exposed to their 
respective reference material for a period of 6 minutes. They then had to respond to 
a PANAS (positive and negative affect scale) questionnaire (Watson, Clark, & 
Tellegan, 1988) which would indicate the momentary affective tendencies (whether 
positive or negative) of the participants in relation to the reference material. 
Subsequently, the respondents performed the design task. The sketching and 
scribbling activities were recorded through screen capture software (Camtasia Trial 
Version) (Fig. 1). 


3.3 Material 


A sample design brief (Fig. 2) was developed which involved the respondents in a 
graphic design task: to produce conceptual sketches of a logo for a fictitious sky 


Table 1 Experimental design of pilot study 


Virtual reality reference R1i—male R3—female 


Screenshot reference R2—female R4—male 
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Fig. 1 Design setup 


Design Brief 


Kindly go through the following information and complete the design task within the given time 
limit. 

PartA 

Design Task: 


Skydiving is a popular adventure sport practiced all over the world. Owing to high demand for 
the sport, a sky diving club named “Deccan high” is being launched. The club intends to promote 
skydiving locally by organizing training programs, sky diving sessions and other such activities. 
The club is formed by a group of sky diving professionals from all over the world and the aim of 
the club is to broaden the community of divers by giving exposure to as many enthusiasts as 
possible. 


You are expected to come up with conceptual sketches for a logo of the club. The logo will be an 
integral part of its brand identity. It will be used in all their promotional material, digital presence 
points like websites, apps and so on. The logo will be part of collaterals like business cards, 
apparel, signage and so on. 


You are free to produce any number of conceptual sketches and explorations. 


Visual references 


You will be given certain visual references of sky diving before you start the design task. You 
must go through the references before starting the design task. 


Fig. 2. Design brief 


diving club named Deccan High. Two sets of reference material (VR and Non VR) 
were also selected. The Virtual Reality reference was a 360° virtual reality video of 
a group of people performing sky diving (Stuykhin, 2016). (User permission 
obtained through personal communication—email dated 24-02-2018). The video 
consists of three segments: on ground/take-off, inside aircraft, skydiving/landing. 
The video was played on a mobile based virtual reality setup (Phone: Redmi Note 
3. HMD: MI VR Play 2). The non VR reference material consisted of screenshots 
gathered from the 360 VR video of the sky diving activity. Screenshots were taken 
on the basis of two factors: change in video segment and change in primary view 
(gaze in different directions, addition/subtraction of elements into and out of the 
frame) of the 360 camera. 
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The decision to employ open access footage such as Youtube videos was driven 
by the rationale that designers/design students generally tend to refer to such freely 
accessible material on the internet in the conceptual design stage. 


3.4 Protocol Study 


Once the task was completed, the method of think aloud protocol study (Cross, 
Christiaans, & Dorst, 1996) was employed to capture cognitive/affective processes 
of the respondents. They produced retrospective verbal protocols by watching video 
recordings of their sketching activity. An interviewer sought clarifications by asking 
questions as and when required. 


3.5 Data Analysis 


A preliminary qualitative content analysis was performed on the video recordings 
of the verbal protocols. The utterances were organized based on categories of 
mental imagery processing derived from ontologies of design cognition (Hay et al., 
2017) and principles of affect-cognition interaction (Clore, Schiller & Shaked, 
2017). Utterances of the respondents which foregrounded the working of the 
principles of affect-cognition interaction (principle of attribution, principle of 
immediacy and principle of affective processing) were isolated. They were then 
analysed to understand the operation of the principles in relation to the given 
reference material, the interaction of affect and mental imagery processing during 
problem solving and the subsequent impact of such processes on the produced 
conceptual designs. 


4 Results 


Both pairs of respondents successfully produced conceptual sketches for the given 
task. Figure 3 shows the final design outcomes of the respondents. While three of 
them explored various iterations (R2, R3, R4) only one respondent (R1) zeroed in 
on an idea early in the design session and continued to refine it. The PANAS 
questionnaires indicated that an overall positive affect (mean momentary positive 
affect = 37.5 and mean momentary negative affect = 17.25) prevailed among 
respondents irrespective of exposure to either type of reference material. 

As indicated in Table 2, the conceptual design task triggered the interaction of 
affect and mental imagery processing in the given conditions. The results indicate 
the working of the principles of affect-cognition interaction in distinct ways 
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Ri 


Deccan #%High 


Fig. 3. Final sketches by the four respondents 


depending on the experimental condition (experimental/control group). In addition 
to this, the interaction of these two modalities of problem solving in turn influenced 
the nature of the final conceptual designs. 


4.1 The Operation of Principles of Affect-Cognition 
Interaction in VR Condition 


Respondents who received the VR reference (RI, R3) revealed through their pro- 
tocols that the central aspect of skydiving was the experience of falling and the need 
for safety in such a sport. They identified skydiving as an intermediate point 
between the sky and the ground and they conferred affective value to the image of 
the parachute as a gear which enables the “safe” experience of thrill. Subsequent 
explorations involving perspective, typography, symbolism and so on (R3) con- 
verged around the identified central aspects mentioned earlier. Though the problem 
statement triggered the image of the parachute as an iconic symbol for the sport of 
skydiving, the additional safety aspect (which generates thrill instead of fear) was 
induced by the VR reference (R1, R3). This indicates that the skydiving VR 
experience generated some degree of immersion as they witnessed the operation of 
the parachute in mid-air as compared to the subjects in non-VR condition who saw 
still images. This clearly indicates the affect-cognition principles at work. Affect 
was attributed to the safety aspect of the parachute (attribution principle). Affect 
was generated by immediately accessible mental phenomena such as the thrill of 
skydiving, the aspect of falling, the need for safety (immediacy principle) and the 
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Table 2 Preliminary classification of utterances of R3 
Verbal protocol and sketch 
“When I read skydiving, the first thing that came to my 


mind was that there is a glider or parachute ...” 
“there had to be a view of a person falling or the earth 
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Cognitive process and principle of 
affect-cognition interaction 
Generation of mental images 

+ 

Attribution, immediacy 


.. wherever I have used circle ... it would denote the 
earth ... like a top view when the person is falling” 


ee |= ED 


“The thing that I had in my mind constantly when I was 
doing any of the iterations is that ... I was trying to not 
make it look like something that would not be like a 
regular, normal view ... I made mental notes of the 
views I was seeing in the reference” 

“T used the reference to note the shape of the 
parachute”; “When I experienced that VR reference the 
safety bit ... was established by that ... 


“T kind of simplified it here...I just kept the Transformation of mental images 
parachute...and the name itself...and then just to like + 

bring it all together I added the circular element...” Attribution, immediacy, affective 
processing 


Maintenance of mental images 
+ 

Attribution, immediacy, affective 
processing 


momentary positive affect acted as “go” signals (affective processing principle) for 
the line of thought that the parachute and the aspect of skydiving as a connector of 
two end points (earth, sky) ought to be included in the logo. This is reflected in the 
style of mental imagery processing: generation (parachute image, the fall from the 
sky to the ground by incorporating perspectives such as bird’s eye view and 
coloured depictions of sky and ground), maintenance (parachute as not just an 
iconic sporting gear but a safety equipment, the fall as a thrilling activity as opposed 
to being dangerous) and transformation (abstraction of parachute image into logo 
designs and fusing the parachute image with the two end points which are the sky 
and ground). 
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4.2 The Operation of Principles of Affect-Cognition 
Interaction in Non VR Condition 


The respondents who received the non VR references (R2, R4) revealed through 
their protocols that they explored various ideas for the logo. Form explorations 
around visual components such as aircraft wings (R2), human figures (R2), 
typography (R2, R4), skydiving formations (R2) posture of divers (R4), atmo- 
spheric elements such as clouds (R4), symbolisms such as images of parachute (R4) 
were attempted by the respondents. When compared to subjects in the VR condi- 
tion, the respondents in the non VR condition generated iterations driven by varied 
aspects as opposed to one or two central aspects. The respondents relied on 
memories such as sky diving scenes in films (R2), sky diving videos (R4) and also 
the visual references (screenshots) (R2, R4) to generate solutions. This indicates a 
markedly different functioning of the affect-cognition principles when compared to 
the VR group. Since the iterations were influenced by the assessment of the 
respondents as being feasible for a logo or not, affect was attributed to their own 
designs as opposed to a central idea or theme (attribution principle). Affect was 
generated by immediately accessible memories and visual references (immediacy 
principle) and the momentary positive affect acted as “go” signals (affective pro- 
cessing principle) for the line of thought/iteration which the respondents thought 
were most feasible for a logo of a skydiving club. This is again reflected in the style 
of mental imagery processing: generation (images generated around various aspects 
of sky diving such as aircraft, divers), maintenance (multiple iterations of design 
concepts based on perceived potential as logo) and transformation (abstraction/ 
fusion of one design iteration into a logo form). 


4.3 Indication of Affect-Cognition Heuristic in Conceptual 
Design 


Table 2 denotes a detailed instance of the verbal protocol of R3 in the form of an 
affect-cognition heuristic. 


5 Discussion and Conclusion 


In this paper, the role of affect has been understood in a designer oriented manner. 
The paper details the need to capture affect in protocol studies to get a more 
comprehensive view of design problem solving processes. The experimental pilot 
study produced results which illustrate how affect influences cognitive aspects such 
as mental imagery processing and in turn the nature of ideation and conceptual 
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designs. The impact of using tools such as immersive mood boards and virtual 
reality references on the affective engagement of designers/design students has also 
been demonstrated. 

The paper foregrounds the interaction of affect and cognition in design problem 
solving. While existing protocol studies tend to focus mainly on cognitive aspects of 
design thinking, this study produces preliminary results which show that cognition is 
not an isolated activity. The salience of affective influence on cognition is much 
greater and is not confined to evaluation alone. The role played by affect indicates 
that designing is not driven by cognitive logic alone. These findings could lead to 
further studies which could offer a holistic view of designer psychology and 
conceptual design. Such studies could enhance existing theories, models and 
ontologies of design. The study also shows how incorporating VR reference material 
into the early stages of design could manipulate the affect-cognition dynamics of a 
designer. Specific formats of reference material elicit different kinds of psycholog- 
ical responses from designers. Depending on the nature of the problem being 
addressed, appropriate reference material may be employed to induce different kinds 
of problem solving modes. Further studies with bigger sample sizes, varied sample 
population, a broader range of design tasks and more dynamic conditions of 
experiments will be undertaken to refine the results of the present study. 
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and Rolf Dornberger 


Abstract The development of virtual reality provides opportunities for immersive 
information visualizations and therefore it is expected to facilitate the exploration 
and understanding of data. Hand gesture control enables intuitive interaction and 
thus it is suggested to amplify the level of immersion further. This paper conducts 
and experiment to identify a set of intuitive gestures when interacting with an 
information visualization. Participants are asked to provide hand gestures to given 
information seeking tasks in an interactive data visualization application in virtual 
reality that they did not know in advance. The results are analysed and findings with 
intuitive gestures are communicated and discussed. 
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1 Introduction 


Companies, which are able to manage and analyse big data, are suggested to gain a 
competitive advantage (Espinosa & Armour, 2016), but often businesses struggle 
with data examination (Boyd & Crawford, 2012). Virtual reality (VR) comes with 
new possibilities for the visualization of such information (Tegarden, 1999) and it is 
proposed to enable an easier perception and manipulation of data (Donalek et al., 
2014). Whereas there are already many examples of visualizing data in VR 
(Garcia-Hernandez, Anthes, Wiedemann, & Kranzlmiiller, 2016), interaction in VR 
as well as the navigation of 3D data visualizations remain to be a challenge and 
come with several unsolved problems (Brath, 2014; Vultur, Pentiuc, & Lupu, 
2016). At the same time, not only the representation but also interaction influences 
how well data are perceived (Ball, North, & Bowman, 2007; Raja, Bowman, Lucas, 
& North, 2004). Based on an experiment this paper examines such interactions by 
observing which gestures participants intuitively use to conduct information 
seeking tasks in an information visualization in VR. 

The paper begins with a literature review about the overlapping domains of VR 
and information visualization and touchless gesture interaction, specifically, with 
data. Furthermore, the framework of the Visual Information Seeking Mantra 
(Shneiderman, 1996) is introduced as a guideline for the tasks examined in the 
experiment. Then the method of the experiment is explained and the results are 
presented. Finally, our findings are discussed and implications for future research 
are proposed. 


2 Literature Review 


2.1 Information Visualization in Virtual Reality 


Managing, analysing and understanding data became a crucial part of valid and 
successful business decisions (Chen, Chiang, & Storey, 2012). Although more and 
more complex data is available, businesses often fail to derive valuable information 
because they struggle with data exploration and discovering patterns (Boyd & 
Crawford, 2012). Information visualization can be an approach to digest such 
abstract data in a more understandable way (Bollier, 2010; Van Dam, Laidlaw, & 
Simpson, 2002). Whereas visualization technologies have been applied in many 
different areas of business since years (Tegarden, 1999), VR now enables infor- 
mation visualizations in a more immersive way. This could allow users to become 
part of a virtual environment and ideally would enable them to embody interactions 
(Teras & Raghunathan, 2015). This allows perceiving and manipulating informa- 
tion in an intuitive and immersive way and could therefore improve data explo- 
ration and understanding (Donalek et al., 2014; Garcia-Hernandez et al., 2016; 
Moran, Gadepally, Hubbell, & Kepner, 2015). 
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VR is already applied in scientific visualizations for the representation of 
physically based data (Nagel, 2006), such as in the area of genomics (Adams, 
Stancampiano, McKenna, & Small, 2002), medicine (Shen et al., 2016) or 
astronomy (Wang & Bennett, 2013) and is reported to have facilitated research 
processes (Van Dam, Forsberg, Laidlaw, LaViola, & Simpson, 2000). For example, 
Garcia-Hernandez et al. (2016) conclude that scientific visualization in VR leads to 
“easier understanding and manipulation possibilities for the data, yielding a more 
efficient and complete analysis” (p. 1). Even though information visualization 
represents more abstract, processed data (Bollier, 2010), the found benefits for 
scientific visualization in VR are proposed to be likewise viable for visualizing 
information in VR (Vultur et al., 2016). Based on a visualization of Twitter data on 
a 3D model map, Moran et al. (2015) demonstrated, that VR provides an immersive 
way for information visualization and that “patterns and visual analytics are more 
efficient when working in a geospatial domain” (p. 6). Various other studies report 
that visualizing abstract data in VR can improve the analysis process compared to 
conventional 2D visualizations, for example by reducing errors and faster com- 
pletion of particular analysis tasks (Alper, Hollerer, Kuchera-Morin, & Forbes, 
2011; Cordeil et al., 2017). 


2.2 Touchless Interaction with Data 


The benefits of exploring data in VR come from its enhanced immersion (Raja 
et al., 2004; Teras & Raghunathan, 2015). But not only the representation influ- 
ences perception, also the way a user interacts with such a visualization has an 
impact on how well information is received (Cabral, Morimoto, & Zuffo, 2005; 
Kosara, Hauser, & Gresh, 2003). Efficiency and effectivity of human-computer 
interaction have shown to depend on the used input device (Bachmann, Weichert, 
& Rinkenauer, 2014). Especially touchless, gesture-controlled input devices are 
suggested to allow a more intuitive and natural way of interaction with computers 
(Cabral et al., 2005) and thus could increase the level of immersion (Zocco, Zocco, 
Greco, Livatino, & Paolis, 2015). With interaction schemes directly based on the 
experience of familiar movements (Deller, Ebert, Bender, & Hagen, 2006), 
gesture-based VR applications allow users for example to grab, throw or push 
objects in VR with hand gestures, as the users is used to from reality (““VR—Leap 
Motion Gallery,” n.d.). Hence gesture-controlled VR applications combine both, a 
virtual environment and intuitive interaction in one immersive experience and thus 
could also bring information visualization to a new level. 

Although interaction is considered to have a significant impact on how infor- 
mation is perceived, the focus of information visualization research has been mainly 
on the new possibilities of representation (Yi, ah Kang, & Stasko, 2007). In the 
related area of touchless navigation, Ball et al. (2007) reported that users not only 
prefer physical navigation, it has also outperformed other virtual navigation tech- 
niques, such as joysticks, in respect to comprehension, speed or orientation. 
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2.3 The Visual Information Seeking Mantra 


Interactions with information visualization systems often require a similar set of 
tasks to be performed by the users. Shneiderman (1996) published “Visual 
Information Seeking Mantra” (VISM) framework of seven tasks (overview, zoom, 
filter, detail-on-demand, relate, history, extract) that are usually necessary to handle 
data in effective way. He furthermore stated that “Overview first, zoom, filter and 
then details-on-demand” is a principle that can be found in many visual design 
guidelines and real life applications. This framework is widely accepted in the area 
of human-computer interaction and in data visualization applications and has been 
extended and updated in various studies (Craft & Cairns, 2005). 

In their proposal for an updated VISM 2.0, Stauffer, Ryter, Dornberger, and Hil 
(2016) separated the initial tasks into four main tasks, which are also part of the 
mantra itself (overview, zoom, filter, detail-on-demand) and three support tasks 
(relate, history, extract). Additionally, they differentiated the zoom task into 
zoom-in and zoom-out as these are considered as two different cognitive tasks. 

The VISM has been widely referred to by researchers developing new infor- 
mation visualization applications or interaction methodologies (Craft & Cairns, 
2005), including the domain of information visualization in virtual environments. 
Wiss and Carr (1999) compared the elements of their cognitive classification 
framework for 3-dimensional information visualization with Shneiderman’s tasks 
(Wiss & Carr, 1998) and based an experiment for identifying important factors for 
the usability of 3D information visualization interfaces on the VISM framework. 
Other papers apply the VISM guidelines to the visualization of object-oriented 
software (Maletic, Leigh, Marcus, & Dunlap, 2001) or digital libraries (Borner, 
2000) in VR. Whereas the VISM framework seems to have arrived in the discussion 
about designing virtual environments, no literature could be found that explicitly 
addresses implications of the VISM on the development of touchless gesture 
control. 


2.4 Research Questions 


Seven years after the publication of the VISM, Bederson and Shneiderman (2003) 
envisioned that computers must become “an extension of the user’s body” as 
information visualization interfaces should “enable users to forget they are using a 
computer and think only of the important work they are accomplishing”. We argue 
that intuitive gesture control combined with VR could enable such an immersive 
information visualization. 

The positive impact of VR on the understanding of information visualizations as 
well as the implication of the VISM on VR have been discussed in literature. At the 
same time, the implications of the VISM on the design of touchless gesture control 
has not received much attention. Intuitive hand gesture control could further 
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amplify the immersion level (and its benefits) of information visualization in VR 
and thus should be investigated in greater detail. But are there VISM tasks that can 
be controlled by intuitive hand gestures? This will be examined based on the 
following statement: 


Intuitive hand gestures can be used for conducting tasks of the Visual Information Seeking 
Mantra to handle data in an information visualization application in virtual reality. 


Due to the choice of the Force-Directed Graph application (Sect. 3.1) as the 
platform, for the experiment and with its ensuing limitations outlined in Sect. 3.2, 
this paper investigates three main VISM tasks. The experiment is based on fol- 
lowing research question: What are intuitive gestures to get an overview, to zoom 
and to find detailed information (detail-on-demand) in an information visualization 
VR application? 


3 Methods 
3.1 The VR Application 


For observing what gestures participants intuitively and spontaneously apply to 
interact with data in VR, an experiment based on an existing information visual- 
ization application was conducted. To find an application for a specific scope of 
examining touchless gesture control in VR, all major platforms for VR applications 
were searched and reviewed. Only a few data visualization apps for VR were 
available and only one of them could be controlled with gestures: “Force-Directed 
Graph” (Kinstner, 2016). 

Force-Directed Graph is an experimental data visualization demo application 
created by Zach Kinstner and is available in the Leap Motion community gallery 
(“VR—Leap Motion Gallery”, n.d.). The application visualizes data nodes in the 
form of a force-directed graph. The data nodes are randomly generated and no own 
datasets can be imported. 

At the time of research, the possible interactions were limited to: 


Pushing and hitting nodes 

Grabbing nodes 

Tapping a node (opens detail menu of node) 
Left hand palm menu (opens main menu) 
Looking around in the virtual space. 


The detail menu of the nodes and the main menu furthermore provide various 
options for manipulation, such as changing the color, size or amount of the nodes. 
In our experiment, Force-Directed Graph application ran on Windows 10 with an 
Oculus Rift Consumer Version (“Oculus Rift’, n.d.) in combination with a Leap 
Motion controller (“Leap Motion’, n.d.) for detecting the hand gestures (Fig. 1). 
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Fig. 1 Screenshot from force-directed graph application 


3.2 Assigning VISM Tasks to App Functionalities 


VISM-related tasks are especially important for the seeking and understanding of 
visualized information. Therefore, the experiment aims to test the tasks presented in 
the VISM 2.0 (Stauffer et al., 2016). Due to the lack of fitting functionalities in the 
available VR app, it is decided to assess overview, zoom-in, zoom-out and 
detail-on-demand tasks and to abstain from an assessment of the filter task. The 
selected tasks are matched with the corresponding application functionalities as 
shown in Table 1. 

As there was no function getting an overview of the nodes, the head movements 
(i.e. looking around) is chosen for evaluating the overview task. This is not directly 
an overview of the data, as intended in the original VISM framework, although it 
still represents the related task of getting an overview in a broader sense. 

Even though the app offers a zoom (i.e. scale) functionality over the main menu, 
no direct gesture control has been implemented yet for zooming. Nevertheless, it 
was possible to observe what zoom gestures participants would use intuitively by 
just pretending that there would be one. 


Table 1 VISM 2.0 tasks assigned to the force-directed graph application 


VISM 2.0 task Gesture input (pre-defined) Visual output (pre-defined) 
Overview Head movement Overview of data visualization in 


virtual space 


Zoom-in Select menu/graph/scale (left Zoom in 
hand palm) 
Zoom-out Select menu/graph/scale (left Zoom out 


hand palm) 


Detail-on-demand | Index finger tap select on data | Opens pop-up window with details 


node about the node 
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Data nodes can be opened by tapping an index finger, this action visualizes a 
pop-up window with more detailed information (Fig. 1). This function is used for 
testing the detail-on-demand task. 


3.3 Survey and Experiment 


The experiment followed a strictly structured survey with a fixed order of questions 
and tasks. First, demographic questions and the level of experience with VR and 
gesture based applications were gathered. Then the participants were instructed to 
look around and acquire the overview of the data visualization, and perform basic 
tasks of pushing and grabbing a data node. This not only allowed them to get used 
to the virtual environment, it was a chance to make them believe, that the ensuing 
tasks in fact would result in a feedback if done right. We hypothesized that 
familiarizing the participants with the environment would result in more authentic 
and intuitive reactions as it would have been possible in a hypothetical “what if’- 
scenario. 

The experiment then continued with the investigation of the following VISM 
tasks. The participants could try up to three gestures per task in order to navigate in 
the virtual environment with the lens of visual information seeking (Sect. 2.3). 
These were tested with the following commands: 


1. “Find more detailed information of a data node.” (detail-on-demand) 
2. “Get nearer to the node-network by zooming in.” (zoom-in) 
3. “Make the node-net appear smaller by zooming out.” (zoom-out). 


As some experiments were conducted in English and some in German and due to 
the different knowledge of technical terms, the exact wording had to be varied 
slightly. The results were collected with an online survey tool and were then 
exported to SPSS for the analysis. After the experiment, the descriptions of the 
observed gestures were categorized to get a comparable result. 


4 Findings 
4.1 Characteristics of Study Participants 


The experiment was conducted with a total of 30 participants in the university 
campus. Most the participants were either students (73%) or academic staff (20%) 
with a broad variety of backgrounds such as science of education, law, business 
information systems and others. Mostly participants were in the Age group 18-34 
(96.7%) with one participant in the age group 35-54 (3.3%). In the experiment, 
females had a share of 67%. 
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Fig. 2 Distribution of participants’ experience with VR and gesture control in VR 


As seen in Fig. 2, 23 participants have assigned themselves familiar with VR 
(i.e. watched relevant videos of VR applications, already tried or experienced). In 
contrast to VR experience, 20 participants had no experience with gesture control in 
VR, only 8 participants responded as already tried or experienced. 


4.2 Details-on-Demand 


Initially, the participants were asked to look around, push and grab a node in order 
to familiarize with the environment. All participants, sometimes with an operational 
support of the examiners, completed these tasks successfully. Following the 
introduction phase to test a details-on-demand task, the participants were told to 
find detailed information of a data node (Table 2). 80% of the 30 participants 
managed to find the correct gesture and tapped on a node with an index finger. 
Another popular attempt was to grab the node with one hand (24% of all trials), 
although exactly this gesture has already been used in the introductory task. 


Table 2 Detail-on-demand task 


Most popular gestures* Trial 1 Trial 2 Trial 3 Overall 
Tap node 13 (43%) 9 (52%) 2 (25%) 24 (44%) 
Grab with one hand 9 (30%) 4 (24%) 0 13 (24%) 
Other 8 (27%) 4 (23%) 6 (75%) 18 (33%) 
n 30 17 8 55 


“For a clear overview, only the most popular gestures are shown in the tables 
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Table 3. Zoom-in task 


Most popular gestures Trial 3 Overall 
Finger pinch (in) 10 (33%) 6 (23%) 1 (5.3%) 17 (23%) 
Swim 10 (33%) 1 (4%) 1 (5.3%) 12 (16%) 
Pull with both hands 4 (13%) 7 (27%) 2 (10.5%) 13 (17%) 
Pinch with both hands 1 3%) 4 (15%) 2 (10.5%) 7 9%) 
Other 5 (17%) 8 (31%) 13 (68%) 26 (35%) 
n 30 26 19 75 

Table 4 Zoom-out task 

Most popular gestures Trial 1 Trial 2 Trial 3 Overall 
Finger pinch (out) 7 (26%) 6 (27%) 1 (13%) 14 (25%) 
Swim (backwards) 5 (19%) 1 (5%) 1 (13%) 7 (12%) 
Push with both hands 8 (30%) 1 5%) 3 (38%) 12 (21%) 
Pinch (out) with both hands 3 (11%) 3 (14%) 0 6 (10%) 
Other 4 (15%) 11 (50%) 3 (38%) 18 (32%) 
n 27 22 8 57 


4.3 Zoom-In and Zoom-Out 


When asked to zoom-in, most of the participants either first tried a finger pinch 
gesture like the pinch-to-zoom gesture of smartphones (33%) or an arm movement 
as if they would swim in water (33%) (Table 3). Other trials included pulling the 
nodes toward oneself or a pinch movement using both hands. Furthermore, also a 
zoom-out gesture was tested. Often, the participants only half-heartedly tried to find 
a zoom-out gesture, probably because they already failed to find a with zoom-in 
gesture before. Usually, the participants tried the opposite gestures of the ones they 
just tried for zooming-in (e.g. pushing the net away instead of pulling it) (Table 4). 


5 Discussion 


The conducted experiment demonstrated that there are some intuitive gestures that 
can be introduced for the VISM tasks, in this case for zoom-in, zoom-out and 
details-on-demand. To fully test our initial thesis statement, the remaining VISM 
tasks would have to be tested in further experiments. Especially some of the support 
tasks might not be transformed into gestures, as they require more flexible and 
nuanced ways of interaction. 

Even though the results based on a sample of 30 participants are not represen- 
tative, the experiment uncovers some interesting tendencies that will be further 
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discussed. Certain gestures were well more popular and thus intuitive to more 
participants than others. In particular, tapping on a node to get detailed information 
(detail-on-demand) seemed to be intuitively clear to most of the participants. 

An example for a “natural” gesture that people used frequently was the swim- 
ming movement for zooming-in. Many of the participants tried to literally dive into 
the data. Another very physical approach was to pull and push the node-net to 
zoom-in, respectively -out. At the same time, the also very popular finger pinch 
gesture indicates, that intuition not necessarily represents a physical, “natural” 
gesture. Pinch to zoom is an abstract gesture widely used on touchscreen devices, 
which seems to have heavily influenced the intuitive behavior of humans. 
Unexpected was that people tended to think that tapping a centered node could give 
access to a menu overview. This is contradictory to the idea that each node rep- 
resents a separate data point and we could not identify a parallel function in 
touchscreens or desktop environments. 

In addition, the experience from classical computer desktops and smartphones 
trained the human intuition. As well, science fiction movies might have influenced 
the participants, as many told us that they would “feel like in Minority Report 
(Molen, Curtis, Parkes, de Bont, & Spielberg, 2002)”. 

The findings of this experiment for gesture intuition is notable, considering that 
the most intuitive gesture are not always the best gesture from an ergonomic and 
technical point of view. It might feel very intuitive to dive into data with a swim 
gesture, however at the same time frequent zooming would turn into an exhausting 
physical exercise. Other intuitive gesture might be problematic to detect for sensors 
or could lead to a weak accuracy (Kinstner, n.d.). 


6 Conclusion and Future Work 


Today’s success of businesses depends on the understanding and manipulation of 
data. Literature states that information visualization in VR might facilitate the 
analysis and digestion of data. We suggest, that such VR visualizations could be 
combined with touchless hand gesture control to create a truly immersive infor- 
mation visualization. Based on an experiment, a set of gestures were identified that 
participants used to conduct particular tasks based on the VISM framework. 
Future research should aim for a better understanding of the influence of gesture 
control on immersive information visualization, especially in visualizations with 
higher density than in our experiment. A replication of our experiment should not 
only be based on a larger sample size, but also a broader set of tasks, including the 
remaining VISM tasks, and other existing data visualization applications in VR 
should be examined. Furthermore, it would be interesting to investigate variations 
between different generations, nationalities or socio-cultural backgrounds. Whereas 
there are clear benefits of incorporating intuition into the design of gesture inter- 
action, more parameters need to be considered. Only when the interplay of 
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intuition, technical restrictions and ergonomic consequences are well understood, a 
sustainable gesture standard can be developed. 

Touchless hand gesture control for immersive information visualizations in VR 
is still at its infancy, but it is probable that hand gestures will enable users to “forget 
they are using a computer and think only of the important work they are accom- 
plishing” (Bederson & Shneiderman, 2003). Thus, these gestures might be part of 
the future of interactive information visualizations. 
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Abstract The purpose of this study is to assess the benefits of Virtual Reality 
(VR) as a form of self-managed Pulmonary Rehabilitation (PR) for patients with 
Chronic Obstructive Pulmonary Disease (COPD). Qualitative data were collected 
including one focus group and four one-one interviews at a Health Centre in South 
Cumbria, and one focus group and two one-one interviews at a General Hospital in 
West Cumbria. A total of ten patients participated. Data were analysed using the- 
matic analysis. This study is one of the first to investigate VR for PR. The findings 
contribute to the body of knowledge of VR for rehabilitation, and for PR more 
specifically. The findings indicate that elderly COPD patients using VR for PR are 
more compliant to conducting their exercises because VR motivates them and 
increases their confidence in physical activity and self-management of their con- 
dition. Patients demonstrated significant improvements in strength and mobility, 
and report psychological benefits including satisfaction in achieving their goals and 
completing milestones within the PR in VR programme. Combining these factors 
together contributes to an overall improved quality of life for COPD patients. 
Recommendations for future developments of VR for PR are discussed. 
Furthermore, this study provides important practical implications for health 
professionals. 
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1 Introduction 


Chronic Obstructive Pulmonary Disease (COPD) has a significant negative effect 
on patients’ quality of life and constitutes a substantial social and economic burden 
on the UK’s National Health Service (NHS) (Ogunbayo et al., 2017). COPD 
patients with Medical Research Council (MRC) dyspnea (difficulty breathing/ 
shortness of breath) grade 3 or above (and especially grade 4 and 5) have been 
shown to have clear exercise intolerance, reduced disease-specific health status, 
reduced mood status, and reduced self-reported daily physical activities (Bestall 
et al., 1999). The need for engaging and activating approaches to providing health 
and social care for people with long-term conditions has been recognised (Singh & 
Ham, 2006). New technologies, including Virtual Reality (VR), are rapidly 
impacting on the development of novel rehabilitation interventions in a range of key 
research areas and subject classification including medicine, computer science, and 
engineering (Huang et al., 2016). In the field of healthcare research, VR has proven 
useful and effective because of its customisation, interactivity, and ability to 
improve learning and rehabilitation rate (Huang et al., 2016; Rose, Nam, & Chen, 
2018). When compared to real-world rehabilitation, WVR-based Pulmonary 
Rehabilitation (PR) has been found to be more interesting and less dangerous 
(Huang et al., 2016), and could assist patients with simultaneously improving 
balance, coordination, muscle strength, and range of motion while providing 
physiotherapists with quantitative data (Matijevi¢ et al., 2013). However, a thor- 
ough exploration and evaluation of VR’s use for Pulmonary Rehabilitation (PR) is 
required (Huang et al., 2016). Therefore, this exploratory study aims to assess the 
benefits of VR as a form of self-managed PR for elderly COPD patients. 


2 Literature Review 


2.1 Current Treatments for Pulmonary Rehabilitation 


COPD can be defined as “a multi-factorial progressive chronic lung disease that 
causes obstruction in air flow” (McCarthy, Casey, Devane, Murphy, Murphy, & 
Lacasse, 2015, p. 1). This obstruction leads to persistent and progressive breath- 
lessness, productive coughing, fatigue, and reoccurring chest infections (Gold, 
2014). Patients with very severe COPD experience disabling physical symptoms 
(e.g. breathlessness), psychological distress, social isolation, and co-morbidity 
(Bourbeau, 2009; Habraken, Willems, de Kort, & Bindels, 2007; Williams, Bruton, 
Ellis-Hill, & McPherson, 2007). Physical activity is unpleasant because of air 
trapping and increased hyperinflation in the lungs, which results in increased 
breathlessness due to subsequent inefficient breathing (Dias et al., 2013; O’Donnell 
et al., 2007). This then leads to anxiety, further breathlessness, exacerbation of 
COPD symptoms, and panic (McCarthy et al., 2015). Therefore, physical inactivity 
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is common in people with COPD, however, this causes muscle de-conditioning and 
reduces the capacity to engage in physical activity (Bourbeau and van der Palen, 
2009; Spruit et al., 2013). Resistive exercises for respiratory muscles and general 
physical exercises are often involved in the care and treatment and part of a PR 
program (Dias et al., 2013). PR programmes often include nutrition guidelines, 
guidelines on medication, as well as physical training methods and respiratory 
muscle (Zaini & Negrini, 2011). The basic goal of PR is to (1) improve symptoms, 
(2) restore functional capabilities, and (3) enhance overall quality of life (Sharma & 
Singh, 2011). 

To achieve the goals of the PR program, patients should adhere to the program 
with monitoring and attendance at least three times per week for several months, 
however, often patients (especially MRC 3-5) cannot consistently perform PR for 
professional reasons, limited mobility, and lack of economic motivation to present 
themselves at the rehabilitation centres as often as necessary (Dias et al., 2013; 
Holland et al., 2016). In the UK, more than 15 million people have a long-term 
condition (LTC) and therefore heavily rely on health services, accounting for at 
least 70% of all NHS spend (NHS, 2017). Home-based (i.e. self-managed) PR 
provides a viable alternative to clinic-based PR programmes (Liu et al., 2008). 
Home-based PR programmes represent effective therapeutic intervention approa- 
ches for relieving COPD-associated respiratory symptoms, re-establishing and 
improving functional disability, enhancing participation in everyday life, promoting 
autonomy, and improving HRQoL (Liu, Tan, Wang, & Chen, 2014; Spruit et al., 
2013). Self-management education of COPD patients has been used to help them 
increase their physical activities and obtain more effective management of their 
disease (Dinesen, Huniche, & Toft, 2013). Additional benefits include improve- 
ments in well-being through enhancement to physical and psychological status, 
time convenience and flexibility in training, cost effectiveness, and patients feeling 
stronger and achieving goals (Lahham et al., 2017; Mendes de Oliveria et al., 2010). 

Studies have focused on several self-management approaches for COPD 
including telerehabilitation (i.e. the delivery of rehabilitation through information 
and communication technologies) using telephone calls (e.g. Dinesen et al., 2013; 
Holland et al., 2013), comparing video and instruction booklet (e.g. Moore et al., 
2012), and mobile applications (e.g. Hardinge et al., 2015; Farmer et al., 2014; 
Williams, Price, Hardinge, Tarassenko, & Farmer, 2014). For example, Williams 
et al. (2014) explored the use of mobile health applications with COPD patients 
aged 50-85 years, and found that patients had increased awareness of the vari- 
ability of their symptoms following the use of the mobile application. In addition, 
using the mobile application assisted with their self-management approach 
regardless of the previous knowledge and experience with mobile applications. 
Similarly, Farmer et al. (2014) focuses on the use of a mobile telehealth application 
through an Android tablet. Patients were allocated a tablet for home use and could 
complete a daily symptom diary and record clinical symptoms using a 
Bluetooth-linked pulse oximeter. The aim of the trial was to investigate the extent to 
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which a telehealth platform designed to support self-management improves quality 
of life for COPD patients. The researchers’ subsequent study (Hardinge et al., 2015) 
found patients increased compliance. Moore et al. (2012) compared the effective- 
ness of home-based PR using two control groups including a video programme and 
an educational booklet. Their findings suggested that participation in a home 
exercise video program could benefit people with COPD more so than the educa- 
tional booklet because there was an improvement in fatigue, emotion, and reduced 
breathlessness in comparison to the other group (Moore et al., 2012). However, 
existing telehealth systems often have restricted times of data entry rather than 
being fitted with individual patient lifestyles, with frequent data errors requiring 
repeated entry (Hardinge et al., 2015). Moreover, studies have begun to investigate 
VR for rehabilitation and it is now considered a useful addition to rehabilitation 
tools in the process of some diseases (Camporesi, Kallman, & Han, 2013; Dimyan 
& Cohen, 2011; Kwakkel, Kollen, ven der Grond, & Prevo, 2003; Wyller, 1997). 
Studies have demonstrated its usefulness for home-based rehabilitation (e.g. Rizzo 
& Kim, 2005; Rizzo et al., 2001), although few studies have explored VR 
specifically for self-managed PR. 


2.2. The Use of Virtual Reality for Rehabilitation 


The capacity of VR to create controllable, multisensory, and interactive 3D envi- 
ronments, within which human behaviour can be motivated and measured, offers 
clinical and assessment treatment options that were not possible using traditional 
methods (Rizzo et al., 2011). Relevant simulated environments can be created 
where assessment of cognitive, emotional, and motor problems can take place 
(Rizzo et al., 2011). The key characteristics of VR including immersion, feedback, 
and interactivity (Brey, 2008) within the virtual environment are expressed to 
varying degrees in rehabilitation-specific systems (Levac et al., 2016). In addition, 
as VR is mobile-based it can be conducted anywhere at any time and it is not 
limited by the confinements of the patient’s home. Indeed, research has found that 
VR is useful for a wide range of health conditions and clinical applications, 
including pain management, physical rehabilitation, treatment of anxiety disorders 
and post-traumatic stress, among many other physical, neurological, and psycho- 
logical conditions (Powell, Rizzo, Sharkey, & Merrick, 2017). Although virtual 
rehabilitation has been the focus of considerable research for many years, its use for 
rehabilitation was limited by the lack of inexpensive, easy-to-use systems that 
promote the user of valid movement patterns (Levin, Weiss, & Keshner, 2015). 
More recently, the increase in interest and its use in a variety of clinical and home 
settings has increased due to the availability of consumer-ready VR hardware and 
software developments (Levin et al., 2015; Powell et al., 2017). 
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In the health sector, the aim of emerging technologies such as VR is not to 
replace clinical expertise and human factors, but rather to provide a set of tools to 
supplement and support rehabilitation and to empower patients to participate in 
their own well-being (Powell et al., 2017). Technology-enhanced rehabilitation is 
beneficial given its ability to individualise treatment programmes dynamically, with 
either intelligent algorithms which respond to patient performance, or customisable 
programs which allow clinicians to create individualised treatment plans within VR 
software (Powell et al., 2017). In addition, health practitioners can access patients’ 
physiological metrics such as breathing, heart rate or activity level during reha- 
bilitation from remote settings, which can then be streamed into the VR system and 
used to dynamically change the user experience (Powell et al., 2017). The realism 
of VR environments allows patients the opportunity to explore independently, 
therefore increasing their sense of autonomy and independence in directing their 
own therapeutic experience (Kizony, Raz, Katz, & Weiss, 2005). In the UK, an 
estimated 1.2 million people are living with COPD. This group are mostly aged 
40 years and over, with the proportion of COPD patients increasing markedly with 
advancing age (British Lung Foundation, 2018). Research has found that elderly 
patients that exercise using VR benefit from improvements in mobility (Bisson, 
Contant, Sveistrup, & Lajoie, 2007; Maillot, Perrot, & Hartley, 2012), muscular 
strength of lower limbs (Jorgensen, Laessoe, Hendriksen, Nielsen, & Aagaard, 
2013), cognition (mainly of executive functions) (Maillot et al., 2012), balance 
control (Bieryla & Dold, 2013; De Bruin, Schoene, Pichierri, & Smith, 2010; 
Pluchino Lee, Asfour, Roos, & Signorile, 2012; Rendon et al., 2012; Szturm, 
Betker, Moussavi, Desai, & Goodman, 2011), reaction time (Bisson et al., 2007), 
and fall prevention (de Bruin et al., 2010; Duque et al., 2013; Schoene et al., 2013). 
Given its benefits, VR could be considered a viable method for rehabilitation, 
although further investigation is required specifically focusing on its use for PR. 


3 Method 


3.1 Participant Profile 


Ten elderly COPD patients, including 4 females and 6 males aged between 62 and 
76, participated in this study. This particular group (MRC 4—5) have significant 
exercise limitation and are at most risk from COPD exacerbation and subsequent 
hospital admission (Healthcare Quality Improvement Partnership, 2018), and 
therefore tend not to attend traditional PR programmes. P1—P10 represents the 
participant number, which is used for the analysis (Table 1). 
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Table 1 Participant profile 


Participant no. Age Gender 
M 
P2 68 M 
P3 69 M 
P4 73 F 
P5 62 F 
P6 71 M 
P7 76 M 
P8 69 F 
P9 63 F 
P10 63 M 


3.2. PR in VR Application 


Within the “PR in VR” programme there are eight modules, one for each week of 
the eight week course. The first module began with an introduction to COPD and 
the PR in VR programme. Modules two to seven consisted of a set of exercises with 
instruction in the form of a digital avatar. Throughout those modules patients were 
provided with information on COPD. The eighth and final module summarised the 
program and provided patients with an incentive for taking part. 


3.3. Experiment 


Patients used the PR in VR programme at home for the entire duration of the eight 
week course. Given that patients could use the programme at their own pace, each 
patient had reached a different module within the PR in VR programme based on 
their physical wellbeing and ability to complete the exercises. This meant that some 
patients had finished all eight modules, while others were repeating certain mod- 
ules, or had only completed the first module. 


3.4 Qualitative Data Collection 


Two focus groups and six one-one interviews were conducted in June 2018. The 
first focus group, consisting of three COPD patients and each of their partners, and 
four one-to-one interviews were conducted at a health centre in South Cumbria. 
One of the one-to-one interviews was an update from a patient who participated in 
the first focus group. The second focus group, consisting of two COPD patients 
only, and another two one-to-one interviews were conducted at a general hospital in 
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West Cumbria. The focus groups were conducted by two researchers, one phys- 
iotherapist, and one healthcare assistant at both locations. The one-to-one inter- 
views were conducted by one physiotherapist and one healthcare assistant in each 
location. The physiotherapist and the healthcare assistant at each location had been 
assisting the patients throughout the course of the eight week programme by vis- 
iting patients’ homes and communicating via phone and text message. Questions 
explored patients’ PR in VR experience including the benefits and areas for 
improvement, patient satisfaction and perceptions, immersive learning, the usability 
of the VR device and application, and future intentions to conduct PR in VR. 


3.5 Data Analysis 


Thematic analysis was employed to identify emergent themes within the data set. 
Overall, nine key themes were identified including seven benefits: increased 
strength and mobility, compliance, motivation, quality of life, improved 
self-management, satisfaction, and confidence, and two recommendations for 
improvement: personalisation and technical capabilities. 


4 Findings and Discussion 


4.1 Benefits of PR in VR 


4.1.1 Increased Strength & Mobility 


As a result of using the VR application, several patients (P1, P3, P4, P5, P7, P8, P9, 
P10) reported an increase in strength, in particular in their legs, which has had a 
positive impact on their mobility and confidence to conduct daily activities. This is 
evident as P8 reported, “I know certainly my legs are stronger, I can feel that even 
just getting up off the chair ... I have taken back more control again with things 
around the house”. Similarly, when conducting daily tasks P9 stated, “after using 
VR, I am less breathless ... my recovery time is much quicker”. In addition, P1 
reported, “I am a lot more flexible, and without the pain”, and PS stated, “my 
walking distance had improved”. 


4.1.2 Compliance 
According to the findings, PR in VR increases patients’ (P1—P10) compliance with 


their exercises. When comparing PR in VR with traditional methods, P9 stated, 
“VR is more akin to my needs ... it fits in more with my routine based on whether I 
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am having a good or bad day”. P4 even reported using VR more than once in a day, 
“IT can use it twice a day when I am feeling good. I built myself up and I started 
using the weights I have at home’’. In line with this, patients enjoyed the flexibility 
of completing their exercises using VR at any point throughout the day (P1, P2, P3, 
P4, P8, P9), “I have done [VR] early in the morning and late at night, around 10 
o’clock” (P1). Having said that, P4, P5, and P7 agreed that PR in VR could 
complement rehabilitation classes at the venue, whereby patients’ build up their 
fitness through VR until they are fit enough to attend the class, which they enjoy for 
socialising. Finally, when comparing VR to using the exercise booklet at home, P8 
stated “I found VR easier than having the booklet at home”’. 


4.1.3 Motivation 


As a result of using PR in VR to conduct their exercises more regularly, patients 
(P1—P5, P7—P10) stated that they are more motivated to leave the house, which 
previously they could only do to a limited degree. “I think I had given up. I decided 
that the illness had gone so far, I was not going to get better. I was not going to go 
out again or enjoy life. It has completely reassured me in that way, and given me the 
energy to get going again” (P8). In addition, knowing that the patient usage and 
data is being tracked by the health practitioners was an important factor relating to 
three patients’ motivation (P2, P4, P5, P9). For example, P5 stated, “it is nice to 
know that someone has their eye on you, it helps you along”, and P9, “I feel more 
secure”. 


4.1.4 Confidence 


The majority of patients (P1—P5, P7—P10) discussed having increased confidence 
with conducting daily activities. For example, “it builds up my confidence because I 
can go out more” (P4), and, “I feel confident enough that I started filling my own 
flasks again” (P7), which also demonstrates how PR in VR has helped patients gain 
back more control in their life. In addition, monitoring their own oxygen level and 
heart rate while exercising was important for increasing patients’ confidence while 
exercising (P2, P5, P6, P8, P9), “it reassured me being able to see my oxygen level 
and heart rate while I was exercising” (P8). Further, patients that reached the final 
module where largely interested in accessing a more advanced level or additional 
programme, therefore, indicating how increased confidence leads to increased 
motivation and desire to continue with the programme. An example of this is that 
P2’s partner enquired, “now that he has done level eight and he is quite happy doing 
it, can he have another programme?”. 


Pulmonary Rehabilitation in Virtual Reality for COPD Patients 285 
4.1.5 Quality of Life 


PR in VR has improved patients’ (P1—P10) quality of life which is evident given 
that one patient (P8) had not left the house in seven weeks, however, since using 
VR the patient is now out socialising and enjoying daily activities. “[I feel] a lot of 
progression. From sitting on the sofa I have been able to go with my husband for a 
coffee, go out in the car ... it had got me to the stage again where I felt I could go 
out and go shopping ... I can go out more now because I can walk further and I 
enjoy it more because I feel better. It is motivating to continue” (P4). In addition, 
patients physical and mental state has improved which motivates them to be more 
active. In support of this, “I feel better physically and ready to do something, to go 
somewhere, more active, after I have done my exercises” (P4). 


4.1.6 Improved Self-management 


For some patients (P1—P5, P8, P9), PR in VR served as a reminder to conduct their 
exercises more often, more effectively, and made exercising “pleasurable rather 
than chore-like” (P9), therefore providing patients with increased enjoyment and 
better self-management of their condition. “I think I had forgotten to change 
positions from being hunched down and not breathing. Now, I sit up, take a deep 
breath and do the Thai Chi exercises. I noticed I was not doing that before” (P8). 
Although P1 was conducting exercises prior to using PR in VR, he reported that PR 
in VR has taught him to conduct the exercises “better and slower”. In addition, “it 
has given me back control to manage my condition. I am confident enough to say 
that I can manage it” (P8). 


4.1.7 Satisfaction 


Further, patients (P1, P4, P8, P9) felt a sense of satisfaction after completing the 
modules and consistently exercising and monitoring their health. For example, P9 
stated, “[what I enjoyed most is] actually achieving the programme, getting from 
start to finish because the first two times I couldn’t complete it ... when I eventually 
go to the end it made me feel better, I could feel the progress”. In addition, P4 
stated, “I feel satisfied that I motivated myself to do it. It did help me to do it. I feel 
satisfied when I sit down and think that’s me done for tonight”. Further, “what I 
enjoyed most was knowing that I could get through the exercises. Knowing that my 
oxygen was not dropping. It was a real achievement for me” (P8), and “when I’ve 
finished the Thai Chi I do feel physically and mentally relieved” (P1). 
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4.2 Recommendations for Future Development 


4.2.1. Technical Capabilities 


Regarding the usability of the headset, the patients found it fairly easy to use (P1— 
P10), “I picked it up quickly. It was not difficult to use” (P5). However, the findings 
indicate that stated that certain technical capabilities could be improved. For 
example, P1—P10 reported the device manufacturing issue whereby the camera 
moves to the right and takes some time to return to centre. Additional suggestions 
for future development are a more lightweight VR headset (P4, P5, P8, P9), and 
“what I want to be able to do is fast forward it and pause it” (P1), which was echoed 
by P2 and P3 and supported by P4 and PS, as this would allow them to complete the 
modules at their own pace. 


4.2.2 Personalisation 


In line with this, given that patients have varying levels of COPD, it is important to 
provide personalised programmes to ensure patients are being challenged based on 
their ability. This is evident given that P2, P5, and P10 found the modules too easy, 
“T feel that the first two were too easy” (P5), and require more advanced levels, as 
this would “make it more attractive and interesting” (P10). In line with this, Pl 
stated, “for people less breath affected, they might want a harder programme”. 
Further, P4 and P6 found the pace “too slow for me” (P6). Whereas P1, and P3—P5 
found the more advanced levels required more practice and enjoyed the flexibility 
to remain on certain levels, “it is beneficial to have the different levels so I can drop 
down depending how I feel each day” (P4). In addition, P5 agreed with P4 and 
added that various exercises within each module would maintain engagement, “I 
would like two or more routines in each module instead of doing the same thing all 
the time. For longer than six weeks it would become boring” (P5). 


5 Conclusion 


The present study aimed to provide initial insights into the benefits of VR for 
self-managed PR by using an exploratory qualitative technique. The findings offer 
initial evidence to support VR’s potential in this specific context by identifying 
seven benefits of VR for COPD patients. The majority of patients in this study 
reported they had not been exercising at home or attending traditional classes, 
however, some had prior experience with the latter. This study demonstrates how 
PR in VR increases patients’ compliance with their exercises, particularly those that 
find it difficult to consistently attend traditional classes due to ill health, lack of 
motivation, suffering from depression, or inconvenience with timing and limitations 
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with transportation. This is because patients benefit from increased motivation, 
confidence, and significant improvements with their strength and mobility. They are 
able to better self-manage their condition, and feel psychological benefits such as 
satisfaction from achieving personal goals and feeling healthier. Together, those 
factors improve patients’ quality of life, therefore, demonstrating the potential of 
this innovative technology as a credible alternative to traditional PR programmes, 
particularly for the target group considering their limitations. 


6 Theoretical Contribution and Practical Implications 


Previous studies (Bisson et al., 2007; Huang et al., 2016; Jorgensen et al., 2013; 
Kizony et al., 2005; Lahham et al., 2017; Maillot et al., 2012; Mendes de Oliveria 
et al., 2010; Rose et al., 2018) have addressed the use of VR for rehabilitation and 
found several benefits including increased compliance, improved self-management, 
increased mobility and strength, and achieving goals. However, to the best of the 
researchers’ knowledge this is one of the first studies to investigate the benefits of 
VR for self-managed PR. Furthermore, this study provides important practical 
implications for healthcare professionals on the benefits of emergent technologies 
such as VR for PR. This innovation differs from any other self-managed pro- 
grammes by giving the health practitioners the ability to supervise patients remotely 
and at scale, and to performance measure delivering Digital Precision Medicine. 
Therefore, the NHS could benefit from less expenditure on physiotherapists 
delivering classes in hospital outpatient departments. Indeed, the national shortage 
of physiotherapists means places are limited and patients are on a long waiting list 
to attend (NHS Funding, 2018), and during this time patients have reported 
non-engagement with exercising. PR in VR provides a solution to this problem, and 
because patients’ data is tracked and observed in real-time, this limits the need for 
regular hospital visits, therefore, reducing costs and time for both the patient and the 
NHS. In addition, patients could feel more confident knowing their exercises are 
being monitored. 


7 Limitations and Future Research 


The first limitation is related to the selected sample. COPD is common among 
people aged 40 years and above, however, this study employed a sample of patients 
aged 62 and over. Therefore, future studies could employ a sample with varying 
demographics (i.e. age range) in order to ensure the findings can be considered 
relevant to the majority of COPD patients. Furthermore, this study outlines only the 
exploratory qualitative phase and future studies will provide statistical evidence to 
support the benefits of VR for self-managed PR in order to advance this research 
area. 
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Exploring Surgeon’s Acceptance Mm) 
of Virtual Reality Headset for Training pa 


Libi Beke Hen 


Abstract Understanding user intention to accept or reject a new technology is a 
key factor in implementation and success of new system technology in an orga- 
nization (Gahtani & King, 1999). The purpose of this study was to explore sur- 
geon’s acceptance of Virtual Reality Headset (VRH) as a surgical simulator. The 
Unified Theory of Acceptance and Use of Technology (UTAUT) model was 
applied to explore the factors that affected surgeon’s acceptance. A structured 
questionnaire was conducted to capture information from international surgeons. 
The result explain 52% of variance associated with Behavioural Intention 
(R2 = 0.521), the three constructs of UTAUT model: Performance Expectancy 
(PE), Social Influence (SI) and Effort Expectancy (EE) have a significance and the 
biggest impact on surgeons behavioural intention to use the Virtual Reality 
Headset, therefore organizations must take into account the following: (1) empha- 
size the usefulness of the software (2) consider surgeon’s psychological aspects. 
(3) The system must be easy to use. 


Keywords Virtual reality headset - Simulator - Surgical training - UTAUT - 
Technology acceptance 


1 Introduction 


Medical science is continuing to break new boundaries, while surgical training is 
still somewhere behind; new technology innovation such as Virtual Reality 
(VR) simulators can offer the answer to this actual gap. The simulation is defined as 
an environment that simulates a segment of reality, and allows the student to control 
the variables of the same reality (Eshet & Hammer, 2006). 
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VR are computer based simulators with a designed software, it provides a real 
physical feeling and a real interaction of clinical situations (Gardner & Raemer, 
2008). While VR simulator have the significant advantage to develop learner skills 
and improve quality of healthcare over traditional methods (McGaghie et al., 2015), 
it is not surprising to see iteration of VR in surgical education, it gives surgeons the 
possibility to train outside of the Operation Room (OR) in a safe environment with 
an interactive and realistic model. VR is at the forefront of innovation for digital 
learning. 

In this study, the author has introduced a new surgical simulator that haven’t 
been used yet in the health care environment. The training tool is a new VRH 
called: HTC Vive. 

HTC Vive was released to the market in April 2016, “VIVE is a first-of-its-kind 
VR platform developed by HTC and Valve for total immersion in virtual worlds. 
Using a Headset and wireless controllers, the learner can explore and interact with 
VR experiences, apps and games that blur the line between imagination and reality” 
(HTC Vive, 2017). 

With the financial sponsorship of Johnson and Johnson medical devices com- 
pany, the author and her colleagues used Johnson and Johnson surgical procedure 
on Knee replacement and created an application supported by the HTC Vive. The 
surgeon is able to step inside surgical virtual OR (Fig. 2), explore proper surgical 
steps and “see” the virtual patient. He can assemble the instruments, use them on 
the virtual patient and also receive a direct feedback while using it. 

As they wanted to evaluate VRH as a potential learning tool and identify the best 
ways to implement it, according to Al-Gahtani and King (1999) the key factor in 
implementation and success of new system technology is the ability to predict and 
identify user’s acceptance. In order to better understand what factors influences user 
acceptance of new technology, researchers have suggested different models 
examining the factors contributing to individuals in the acceptance of Information 
Technology (IT). 

Nowadays, different publication studies are exploring VRH as a training tool in a 
wide range of subjects such as: Driving simulation for accident prevision (Taheri, 
Matsushita, & Sasaki, 2017), student’s behavioural intention to use VRH in 
learning (Shen, Ho, Kuo, & Loung, 2017), and recent study by Pulijala et al. 2017 
explore the effectiveness of VRH in surgical training on surgical resident. 

The motivation to conduct this research was the lack of empirical support 
assessing surgeon’s intention to use VRH. The author has used the “Unified Theory 
of acceptance and use of technology” (UTAUT) model (Venkatesh, Morris, G. 
Davis, & F. Davis, 2003) to develop a structural model of surgeon’s acceptance of 
VRH simulator, data was collected from orthopaedic surgeons from all over the 
world with different level of surgical experience from age 30-65. The author has 
made an empirical contribution to the UTAUT research by conducting the first 
research on surgeon’s acceptance of VRH as educational simulator. 

The key research question that the study was designed to explore is: What 
critical success factors/determinant will influence surgeon’s intention to use VRH 
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simulator? Next: Are Surgeons willing to use the VRH simulation for training 
purpose? Do surgeons perceive the VRH simulation to be free of effort? 

Finally: Do surgeons perceive the VRH simulation to be useful as part of the 
training experience? 

The findings of this study provide a useful guidance to health care professional 
trainers, medical devices industry and hospitals, assessing the integration of VRH 
simulator such as HTC Vive in their organizations, and help them to understand the 
critical success factors that influence surgeons accepting the VRH as an educational 
tool. 


2 Literature Review and Hypothesis 


2.1 Theoretical Framework 


2.1.1 Unified Theory of Acceptance and Use of Technology (UTAUT) 


Venkatesh et al. (2003, 2008) compared empirically the eight models of technology 
acceptance; each of this model explore user behaviour by using variety of factors, 
Venkatesh et al. have unified and integrated the four core determinants of intention 
and usage of new technology system based on the similarity of the eight models 
into new theory names as: “Unified Theory of acceptance and use of technolog” 
(UTAUT), this theory aimed to integrate all the factors that were known to affect 
the user adoption into one model. UTAUT presented by it researchers as the final 
model incorporates what is known and provides an infrastructure guideline for 
future research in the context of the acceptance and adoption of technology by users 
(Venkatesh et al., 2003). Respectively, the validity and reliability of this model for 
adoption of technological systems have been shown in numerous studies and in 
different contexts. (Oye, Iahad, & Rahim, 2014). A group of determinates that 
influence the adoption of new information technology by the user can be divided 
into two groups of factors; The first group consists of beliefs about the system. For 
example, determinates: ‘perceived usefulness’ (PU) and ‘perceived ease of use’ 
(PEOU) of the system represent the user belief about the system (Davis, 1989). 
A second group of factors that influence the acceptance of the information system 
includes; social factors associated with the user social environment so call “social 
influence” (Holden & Krash, 2010). Different studies examining the role that these 
factors play in using the system and the finding shows: users tend to use the system 
more when they believe that people who they care about, or people who have an 
impact on their behaviour, think they need to use the system (Lu, 2014; Williams, 
Rana, & Dwivedi, 2015). 

The UTAUT model (Venkatesh et al., 2003) as indicated in Fig. 1, hold four key 
constructs that influence behavioural intention to use a technology: (1) Performance 
expectancy (PE)—‘the degree to which an individual believes that using the system 
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Fig. 1 Structural model of hypothesis 


will help him or her to attain gains in job performance”. (2) Effort expectancy 
(EE)—“the degree of ease associated with the use of the system’’. (3) Social influence 
(SD)—“‘the degree to which an individual perceives that important others believe he or 
she should use the new system”. (4) Facilitating conditions (FC)—‘the degree to 
which an individual believes that an organizational and technical infrastructure exists 
to support use of the system” (Venkatesh et al., 2003, pp. 447-453). As the model, has 
been tested empirically, three key constructs of the predictors were found as directly 
determinants of user intention and usage, (1) Performance expectancy (PE) was found 
to be the strongest predictor at all points for both mandatory and voluntary setting in 
Venkatesh et al. (2003) study. Performance Expectancy (PE) in the UTAUT model 
captures the nations of: perceived usefulness, motivation, job-fit, advantage and 
outcome expectations. (2) Effort expectancy (EE) in the model is similar to perceived 
ease of use and (3) Social influence in previous models is such as social form and 
subjective norm, were found also having significant influence on behaviour inden- 
tation to use the system but slightly less since it was significant only for the first period 
of usage. The fourth and last predictor (4) Facilitating conditions (FC) represented 
organizational support, which has no significant direct impact on initiation to use the 
information system but had significant direct impact on the actual use of the system. 
Finally, gender, age, experience and voluntariness of use, were posited as moderating 
the impact of the four predictors on usage intention as indicated in UTAUT model 
Venkatesh et al. (2003). 


2.1.2 Advantages of UTAUT 
Unlike TAM that explain 40% of the user acceptance of new technology, UTAUT 


model explain over 70% of variance of usage behaviour (Holden & Karsh, 2010). 
Hence, we can conclude that UTAUT is explaining more factors influencing BI. 
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As well, it combines all eight base models into one unified model (Venkatesh et al., 
2003, pg. 425). 

If we compare UTAUT to TAM, UTAUT includes the social influence which is 
a very important component in today environment, the impact of social media in 
our daily lives is enormous, it become an integral part of our daily activities. TAM 
is a simple model and we cannot except from such a simple model that was created 
in the beginning of the nineties were social media was in it very beginning, to 
explain decisions on acceptance of technology in a wide range of technologies like 
we have today. 

Since UTAUT is the most up to date technology accepted theory that merge with 
other theories, the author has chosen to use it in her study. 


2.2 Hypothesis 


The research model is illustrated in Fig. 1 consistence with UTAUT model. The 
study criterion variable is surgeon’s acceptance of VRH simulator for training, 
which is surgeon’s intention to use and trained with VRH given the opportunity. 

Consistence with UTAUT model, the hypotheses will be presented in two 
relationship categories; one has a direct influence (the independent variables) on 
behaviour intention (the dependent variable), and the second category has indirect 
influence on the behaviour intention to use the new technology by moderating the 
impact of the independent variables. 

As the system, haven’t been utilized yet, the actual usage behaviour cannot be 
measured. In consistence with Venkatesh et al. (2003) study (UTAUT) the 
Behaviour Intention (BI) will be used to predict actual use and the measurement of 
use will not be applied. 


2.2.1 Direct Influence -the Influence of Performance Expectancy (PE), 
Effort Expectancy (EE) and Social Influence (SI) on Behavior 
Intention (BI) 


As stated by Venkatesh et al. (2003) “Performance Expectancy (PE) is the strongest 
predictor determinant of behavioural intention”. 


H1. The greater the Performance Expectancy (PE), the greater will be surgeon’s 
acceptance of the VRH simulator. In other words, PE will have a significant pos- 
itive influence on behavioural intention to use the VRH. 

Effort Expectancy (EE) is similar to ‘ease of use’ from the original TAM, it means 
that the expected use will be free of effort (Davis, Bagozzi & Warshaw 1989). 
Complicated technology that required a- lot of effort will discourage HCP to use it, 
on the other hand, technologies that perceived to be easy will have higher intention 
to be used (Holden & Krash, 2010) the effect of EE on BI among Health Care 
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Professionals was found to be positively associate with BI. (Anja, Heiko, & Ulrich, 
2014). 

H2. The lower the Effort Expectancy (EE), the greater will be surgeon’s acceptance 
of the VR simulator. In other words, EE will have a significant positive influence on 
behavioural intention to use the VRH. 

Social Influence (SI) is define as the perception of an individual that important 
others believe he or she should be using the new technology. The influence of SI on 
BI is significant positive in early stages of experience and affect intention under 
mandatory situations (Venkatesh et al., 2003). 

Although in this study the setting is voluntary, the technology is tested in a very 
early stage of experience and consequently the propose hypothesis is: 

H3. The greater the Social influence (SI), the greater will be surgeon’s acceptance 
of the VR simulator. In other words, SI will have a significant positive influence on 
behavioural intention to use the VRH. 

Facilitating condition is define as the degree to which individual believe he or she 
has the organizational and technical support to use the new technology and that the 
support is design to remove barriers to use (Venkatesh et al., 2003). 

Although this study is not measuring the actual use; 

H4. Facilitating conditions will not have a significant influence on Behavioural 
Intention to use the VRH. In other word, FC will not directly affect acceptance of 
the VR. 


2.2.2 Indirect Influence—The Influence of the Moderating Variables 
on the Independent Variables 


Venkatesh et al. (2003) have identified in their study four key moderating variables: 
age, gender, experience and voluntariness. The influence of the moderating variable 
on the independent variable is presented in 


HS. The influence of Performance Expectancy (PE) on behavioural intention will be 
moderated by age. 

H6. The influence of Effort Expectancy (EE) on behavioural intention will be 
moderated by age. 

H7. The influence of Social Influence (SI) on behavioral intention will be moder- 
ated by age. 

H8. Gender will have no moderated effect on surgeon’s acceptance of VRH. 


3 Data and Methodology 


In this study setting VRH named: HTC Vive was used. HTC vive was released in 
April 2016. 
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The hardware (Fig. 2) is containing two Lighthouse base stations for tracking the 
user movements, two controllers for user interactions and the Headset for immer- 
sive experience. To run the experience, a solid PC must be provided. With very 
high-resolution image given by the Vive Headset, a natural and convenient feeling 
is displayed, the user can look up and to the side with his eyeballs rather than by 
moving his entire head. The user can move around his ‘real-world’ environment 
without removing the Headset. The Headset is very light since it is the computer 
that handles all the heavy graphics. 

The lighthouse base stations are tracking the motions and the users can move 
freely in the room what helps the users feel like he can explore the space. The 
controllers are designed to behave as the user’s hands and are interacting with the 
virtual world—a haptic feedback integrated in each controller, it helps to let the user 
know when he has completed the correct action. 

The author and her colleagues have created an application supported by the HTC 
Vive which allow the surgeon to enter a surgical virtual environment—(Fig. 3)— 
this medical environment is a virtual Operation Room (OR) allowing the surgeon to 


ede 
cm x 


Fig. 2, HTC Vive hardware (HTC Vive, 2017) 


Fig. 3. Left: surgeon in a congress trying the VR simulator using HTC Vive. The screen in the 
background shows the same sequence as appears to the user in the VR glasses (right) 
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perform a full operational procedure and “see” the virtual patient. He can assemble 
the instruments, use them on the virtual patient and also receive a direct feedback 
while using it. 

The surgical procedure presented on the software is a full knee replacement 
training—open surgery, the trainee can perform the all procedure and learn all the 
technical steps. Appropriate guidance is provided in order to stimulate the learning 
mechanism; which tool to pick up from the surgical table and how to place it 
correctly while using it. For the trainee, this environment giving the feeling like he 
is inside the real OR, he can move around and even adjust the lights on the patient 
knee. 


4 Method 


Johnson and Johnson Medical Devices Company, have sponsored the author to 
travel and attend two big Scientific congresses (Fig. 3), one is the EKS (European 
Knee Society) 750 surgeons from all over the world, and the second is EFORT 
(European Federation of National Associations of Orthopaedics and Traumatology) 
10,000 surgeons from around the globe. In the form of these congresses, the author 
has presented the VRH system as an exhibitor. 

As mentioned in previous section, the application presented by the VRH system 
is a medical simulator for knee replacement. 

The VRH simulator attracted a lot of attention and surgeons were coming vol- 
untarily requesting to try this new technology. At the time of the actual use, the 
author has been guiding each participant on the procedure and acted as a trainer. By 
the end of the experience (proximity of ten minutes, since the software was 
designed to stop after ten minutes for exhibition purposes) the study was presented 
and asked the surgeon for his agreement to answer the questionnaire. 

In addition to the congresses, the author has attend a Johnson and Johnson 
course and met with the orthopaedic department of Saint Jean hospital in Brussels. 

The course target audience were; experience surgeons who are using the knee 
replacement product and procedure. The author presented her study following with 
the experience; similarly, to the data collection method in the congresses, the 
questionnaire was handled to the surgeon after trying the VRH simulator. 

Data was collected from March 2017 through June 2017, from all above sources. 
A total of 111 surgeons, have participated in the study. All 111 participants received 
the questionnaire, 4 responses had to be discarded due to invalid data entries, the 
sample comprised a total of 107 participants. 95% of the responses were male and 
aged between 30 to 40 years old (46%) (see Table 1). As the software was 
developed to support the knee replacement product training, no bias has occurred in 
group selection. 

A structured questionnaire was conducted to capture information from the sur- 
geons. All UTAUT predictors, i.e., Performance Expectancy (PE), Effort Expectancy 
(EE), Social Influence (SI), Facilitating Conditions (FC) and Behavioural Intention 
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Table 1 Demographic profile of the sample (N = 107) 


Variable Content 

Gender Male 
Female 

Age 30-40 
41-50 
51-60 
>60 

Involved in training Yes 
No 

Previous experience with VR Yes 
No 


(BI) were measured by used multi item, 5-point Likert-type scales [(1) Strongly 
disagree, (2) Disagree, (3) Neutral, (4) Agree, (5) Strongly Agree]. The questionnaire 
is presented in Appendix I. All items were sourced from the original UTAUT model 
with a small adaptation to the study settings. 

The author collected data on age, gender, year of practice, degree of interest in 
new technology, degree of involvement in training and development and asked if 
the participant had previous VRH experience. Furthermore, the author and her 
colleagues were keen to know if the surgeon will recommend VRH training to other 
surgeons and if he could think in investing into VRH, therefor, two questions were 
added at the end of the questionnaire. 


5 Analysis and Results 


Same as in the original UTAUT study (Venkatesh et al., 2003) partial least squared 
(PLS) was used to analyse the data and SmartPLS as software. 

Structural Equation Model (SEM) is examining relationship among variables 
while including two sub models in the structure such as: Inner model and Outer 
models. The Inner model is also known as the structural model and should specifies 
the relationship between the constructs (PE, EE, SI, FC) to Behaviour Intention and 
to actual use. The Outer model is also known as the measurement model which 
specifies the relationship between the constructs and gives an overview on how the 
constructs are measured. 

PLS is useful for Structural Equation Model (SEM) as it focusses on the analysis 
of the variance and it has been proven to be a good analysis technique for small 
sample size. (Wong, 2013). 

Most of the items (indicators) found to be reliable as the loadings were above the 
recommended 0.7, except items EE3 and SI2 (Table 2), Indicator EE3 was mea- 
sured in a reversed way, to test the attention of the responders, hence, before import 
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Table 2 Model’s reliability and validity 
Construct Indicators Loadings | Composite | Average 
(latent reliability variance 
variable) (CR) extracted 
(AVE) 
Performance | (PE1) Using the virtual reality would 0.888 0.861 0.676 
expectancy enhance my effectiveness on the job 
(PE) (PE2) I would find the virtual reality 0.709 
useful for training 
(PE3) Overall, I believe that the virtual 0.858 
reality is easy to use 
Effort (EE1) Learning to use the virtual Reality | 0.874 0.858 0.671 
expectancy simulator is easy for me 
(EE) (EE2) Overall, I believe that the virtual | 0.878 
reality is easy to use 
(EE3) It takes too long to learn how to | 0.690 
use the virtual reality to make it worth the 
effort 
Social (SIL) My management supports new 0.866 0.795 0.568 
influence technology in training 
(SD (SI2) I would use Virtual Reality 0.641 
simulator if my friends and colleagues 
use it 
(SI3) Hospitals and industry partners 0.736 
encourage surgeons to use VR simulators 
Facilitating (FC1) I know the resources necessary to | 0.733 0.847 0.651 
conditions use virtual Reality 
(FC) (FC2) I have a technical team in the 0.777 
hospital that can assist me with virtual 
reality difficulties 
(FC3) I feel comfortable introducing a 0.902 
virtual reality simulator in my hospital 
now 
Behavioral (BIJ) I intend to use virtual reality again, | 0.900 0.915 0.782 
intention given the opportunity 
@D (BI2) I predict I would use the virtual 0.880 
reality, given the opportunity 
(BI3) I can imagine using virtual reality | 0.872 
regularly for training purposes 
Age*PE Age*PE1 0.757 0.784 0.573 
Age*PE2 0.407 
Age*PE3 0.989 
Age*EE Age*EE1 0.748 0.858 0.669 
Age*EE2 0.817 
Age*EE3 0.674 
Age*SI Age*SI1 0.777 0.817 0.599 
Age*SI2 0.711 
Age*SI3 0.849 
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to SmartPLS the indicator was reversed to fit the Likert format similar to the other 
indicators (=6—response value), this might be the reason for unreliable item. 
However, EE3 and SI2 were retained since the loading is very close to 0.7. 

Composite Reliability (CR) was used to evaluate how well constructed are 
measured by their items (indicators), all CR values in the model are larger than 0.7 
(Table 2), therefor, a high level of internal consistency reliability have been 
demonstrated among all constructs. 

Convergent validity was assessed by evaluating Average Variance Extracted 
(AVE), it is found that all the AVE values exceed 0.5 (Table 2), so convergent 
validity is confirmed. 

Discriminant Validity was assessed according to Fornell and Larcker (1981), 
when the square root of AVE in each construct (latent variable) is larger than the 
correlations between the constructs then discriminant validity exists. 

Looking at the inner model path coefficient it is suggested that Performance 
Expectancy (PE) has the strongest effect on Behavioural Intention (0.469), followed 
by Social influence (SI) (0.233), and last Effort Expectancy (0.155). Facilitating 
Conditions had the lowest effect on Behaviour Intention (0.023). For PE, EE and SI 
the hypothesis path relationship is statistically significant. However, the hypothe- 
sized path relationship between FC and BI is not statistically significant. This is 
because its standardized path coefficient (0.023) is lower than 0.1. Thus, we can 
conclude that: PE, EE and SI are moderately strong predictors of BI, but FC does 
not predict BI directly. Furthermore, this finding supports the author hypothesis 
path relationship, and consistent with the original UTAUT model where FC has no 
relationship with Behavioural Intention, hence, it was excluded from the final 
calculation. 

Bootstrapping (1000 resample) was used to measure the path significance, 
testing the inner and the outer model present the result of testing the hypothesis, 
using two tailed T-statistic, with significance level of 5%. The model was controlled 
for age as a moderator. 

As expected and confirmed by author’s earlier finding from PLS-SEM results, 3 
out of the 4 constructs relationships were found significant. The model explains 
52% of variance associate with Behavioural Intention (R2 = 0.521) and 5 
hypotheses out of 8 were accepted. Table 2, describe the correlation coefficient, the 
T-value and the P-value that associate with each relationship in the model. The 
result support H1 and confirm that Performance Expectancy (PE) has a positive and 
the strongest impact on surgeon’s acceptance (t = 3.984, p = 0). H2 and H3 were 
supported and show a significant direct effect of Social Influence (SD (t = 2.132, 
P< 0.05) and Effort Expectancy (EE) (t = 1.997, p< 0.05) on behavioural 
Intention (BI). H4 was supported as Facilitating Conditions (FC) was found not 
significant (t = 0.256, p > 0.05) and confirm the hypothesis and UTUAT original 
model, FC does not influence surgeon’s BI. 

Age was tested as a moderator and did not have a significant effect on PE and EE 
but did have a significant effect on SI, therefore HS and H6 were rejected but H7 
was accepted. H8 was not tested with consideration to the skewness of the gender 
distribution. 
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5.1 Implications and Limitations 


UTAUT model was not originally designed to evaluate Health Care Professionals 
(HCP) acceptance, nevertheless, this study supported the finding of the original 
model: with 52% of variance explaining BI. Furthermore, PE, EE and SI were 
found to be statistically significant predictors of surgeon’s acceptance of VRH, and 
FC was found not significant predictor of BI. 

If Surgeons perceive the VR simulator as useful, it is most likely to increase their 
intention to use it, in other words: surgeons will accept the technology when it is 
considered to be useful to his or her practice. As PE was found to have the strongest 
impact on BI it is consistent with the finding of Holden and Krash (2010), Li et al. 
(2013), Williams et al. (2015), Chau and Hu (2002) and many other studies. 

To conclude; the critical success factor that influence surgeons accepting the 
VRH as an educational tool are: (1) Performance Expectancy is a key, emphasize 
the usefulness and design the software to bring direct benefits to the surgeons in the 
form of improved effective process. (2) Social Influence has an important role 
consider surgeons psychological aspects. Increasing the driving force of peer col- 
leagues to use the VR may affect surgeon’s behaviour, and will increase their 
willingness to use the new technology. (3) Effort Expectancy is crucial, time to 
learn a new system has a consequence on the intention to use. The system must be 
easy to use. Convincing surgeons that VRH simulator such as HTC Vive is low 
effort will be important to ensure they see advantages in using the technology. 

In this study, some limitation should be considered: the sample was only tar- 
geted orthopaedic surgeons on a voluntary basis. The software presented on the VR 
hardware was only focused on the knee replacement procedure, since it was 
developed by Johnson and Johnson. These limitations may have affected the study 
findings, as the result cannot be generalized to other professional groups. 

This study has provided a useful contribution to the UTAUT studies and to the 
medical devices industry as it is the first study on VRH acting as a surgical training 
simulator, targeting surgeons from all over the world. 

Considering the importance of nurses in the OR a future study on HTC Vive 
acceptance by nurses is suggested. Moreover, future study can investigate other 
professional groups. With the same study setting a future study is suggested by 
examining all UTAUT moderating effects and the actual use. The actual use can be 
examined as soon as the HTC Vive simulator will be integrated in the organiza- 
tions. Besides, other setting than voluntary can be compered. 
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Evaluation of Virtual Reality Mm) 
in Orthopaedic Training—A Pioneering crak 
Pilot Study 


Ronnie Davies, Natasha Moorhouse, Timothy Jung, 
Saleem Mastan and Bibhas Roy 


Abstract This study aimed to evaluate the effectiveness of VR for surgical training and 
identify the user requirements for surgical training VR applications. Nine orthopaedic 
surgical trainees were recruited. A multimodal approach was employed using three 
elements of data collection. This included measurements of head movement while 
experiencing VR, followed by a survey, and then a one-to-one interview. During the 
survey and interviews, three key areas were investigated including the technical quality 
of the VR material, VR usability, and the VR user experience. The VR application 
design provided surgical trainees with detailed imagery as well as rich information, 
allowing them to apply theory to practice during remote virtual training. However, 
several recommendations to improve the quality of the video, the usability, and the user 
experience are offered. Moreover, the findings suggested a positive educational impact 
of the VR application, although further evidence is required. Theoretical contributions 
and avenues for further research are offered together with the implications for practice. 


Keywords Virtual reality - Surgical training - Simulation - User experience 


1 Introduction 


The use of Virtual Reality (VR) has been shown to be of benefit for surgical skills. 
However, it seems that simulators and three-dimensional (3D) learning for ortho- 
paedic surgery seem to lag behind other disciplines (Johns, 2014). In the past two 
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decades, there has been limited research into whether orthopaedic simulators have a 
role to improve practice, and even fewer studies exploring how this can be validated 
(Rambani, Ward, & Viant, 2014). Currently, neither simulators nor VR immersive 
videos are available to a large population of orthopaedic trainee surgeons or 
fully-qualified surgeons. The majority of simulated teaching comes from courses 
which need to be self-initiated. Surgical videos provide close-up snapshots of 
operations, but lack the real-life field of view that is crucial for learning how to 
operate. For example, in arthroscopic (keyhole) surgery, the view is often of the 
inside of the joint (from the viewpoint of the camera inside the joint), but it is 
important to simultaneously see where incisions and instruments are placed outside 
the patient. There seems to be a shift towards the use of surgical VR. The previous 
conventional teaching of ‘see one, do one, teach one’ seems to be an inefficient 
method of learning, with no guarantee of there being a varied case mix, quality of 
quantity. There is also a growing focus on competency based medical education, 
which fits in with the restrictions of junior doctor working times. Simulation, and 
VR in theory, provides trainees to practice and view technical skills in an envi- 
ronment which is safe and does not put the patient safety at risk (Akhtar, Chen, 
Standfield, & Gupte, 2014). VR immersive videos can now be obtained using 360° 
cameras which allows procedures to be viewed with a wide field of vision in the 
operating theatre. Furthermore, detailed operative field views from a secondary 
camera can be superimposed to provide greater clarity for some parts of the 
operation. This technology is accessible using standard VR headsets and the trai- 
nee’s own mobile phone. 

This study explores the effectiveness of VR training for orthopaedic surgeons 
and identifies several user requirements for surgical training VR applications. The 
results can be used to optimise the delivery of VR training in surgery. This paper 
begins with a review of the literature focusing on two separate agendas: (1) the 
current use of VR in medical and surgical training, and (2) the user experience of 
VR applications specific to medical and surgical training. The methods and 
methodologies are then outlined which includes a preliminary intervention and both 
qualitative and quantitative methods and gathering of head movement data. The VR 
application design provided surgical trainees with detailed imagery as well as rich 
information, allowing them to apply theory to practice during remote virtual 
training. However, several recommendations to improve the quality of the video, 
the usability, and the user experience are offered which will aid in future devel- 
opment. The findings suggested a positive educational impact of the VR applica- 
tion, although further evidence is required. Finally, the discussion and conclusions 
are presented followed by the theoretical contribution and practical implications. 
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2.1 VR in Medical and Surgical Training 


Laparoscopic surgery is technically demanding and requires specific psychomotor 
abilities and skills that are different from those required in conventional training 
(Figert, Park, Witzke, & Scwartz, 2001) and difficult to obtain in the operating room 
(Scott et al., 2000). The skills required include altered tactile feedback, different 
eye-hand co-ordination, translation of 2D image to a 3D working area, and the 
fulcrum effect (Gallagher, McClure, McCuigan, Crothers, & Browning, 1999). 
Acquiring the skills to become competent is associated with a long learning curve, 
and extensive training is required in order to be able to move instruments within the 
operative field safely and effectively (Alaker, Wynn, & Arulampalam, 2016). 
However, learning such skills can be inefficient, time consuming, and may pose 
safety concerns for patients (Gallagher, McGuigan, & McClure, 2001). 
Traditionally, residents have been trained in classic apprenticeship format, prac- 
ticing basic surgical and laparoscopic skills in the operating room 
(Yiannakopoulou, Nikiteas, Perrea, & Tsigris, 2015), however, it is now unac- 
ceptable to train this way because it exposes patients to potential risk (Alaker et al., 
2016). Therefore, learning technical and non-technical skills outside the operating 
room has become an essential part of surgical training (Nagendran, Gurusamy, 
Aggarwal, Loizidou, & Davidson, 2013; Seymour, Gallagher, Roman, O’Brien, 
Bansal, Anderson et al., 2002), and current methods include physical box training 
and video training while more recent research has begun to investigate VR training 
and VR simulation training. 

VR simulation trainers digitally recreate the procedures and environment of 
laparoscopy (Buckley, Nugent, Ryan, & Neary, 2012). In comparison to 2D 
videography, VR provides an enhanced visual overview of the procedures being 
performed and offers a more realistic experience as if the trainee is in the operating 
room. Research has indicated that training in VR simulators could improve the 
technical skills of surgical trainees (Nagendran et al., 2013). The use of simulation 
training in education has been recognised for some time and in medical education it 
has taken a variety of settings (Gorman, Meier, & Krummel, 1999). Simulation is 
the replication and modelling of real-life situations for training, testing, and other 
purposes such as scenario planning and design verification (H. R. H. Patel & B. 
P. Patel, 2012). It can be any educational program or technology that removes the 
actual patient from the equation to allow novice learning and skill mastery to occur 
in a low-stress, high-feedback environment, while protecting the patient from 
procedural inexperience (H. R. H. Patel & B. P. Patel, 2012). Hence, VR simulators 
provide basic skills training without supervision in a controlled environment that is 
free of pressure of operating on real patients (Yiannakopoulou et al., 2015). The 
skills obtained through VR simulators are transferable to the operating room 
(Yiannakopoulou et al., 2015). In addition, VR simulators produce objective 
measures of performance, allow for real-time feedback to trainees, and is a safe, 
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ethical, and repeatable alternative for early-stage surgical trainees to acquire tech- 
nical skills for laparoscopic surgery (Nagendran et al., 2013). Therefore, in addition 
to physical box trainers and video trainers, VR simulation has become increasingly 
used for surgical training. 

Researchers have begun to compare the benefits of VR training for medical and 
surgical training with traditional methods (e.g. Alaker et al., 2016; Gurusamy, 
Aggarwal, Palanivelu, & Davidson, 2009; Huber et al., 2017; Nagendran et al., 
2013; Yiannakopoulou et al., 2015). For instance, Gurusamy et al. (2009) con- 
ducted a large systematic review exploring the effectiveness of VR training for 
laparoscopic training in comparison with video trainer training, no training, stan- 
dard laparoscopic training or different methods of VR training in surgical trainees 
with limited or no previous experience. The authors concluded that residents who 
were trained with VR perform surgery faster than those with conventional training 
and is at least as effective as video trainer training (Gurusamy et al., 2009). Through 
a meta-analysis, Alaker et al. (2016) found that VR simulation is significantly more 
effective than video trainers and at least as good as box trainers. Similarly, 
Nagendran et al. (2013) found that VR training decreases the operating time by an 
average of ten minutes, and improves the operative performance of surgical trainees 
with limited laparoscopic experience when compared with no training or with 
box-trainer training. Similar reviews have also explored shoulder arthroscopic VR 
(e.g. Miller, Ziegler, Bauer, & Soldner, 1995; Sutherland, Bresina, & Gayou, 
1994). However, it is unknown whether the impact of the decreased time and 
improvement in operative performance on patients and healthcare funders in terms 
of improved outcomes or decreased costs (Nagendran et al., 2013). Therefore, 
further well-designed trails and low risk of bias and random errors that assess the 
impact of VR training on clinical outcomes are required (Nagendran et al., 2013). 


2.2. VR User Experience 


In order for any VR training programme to be effective, it needs to demonstrate 
acceptability, validity, reliability, and reproducibility in the real life operating 
environment (Buckley et al., 2012). One key element in the design of any VR 
system is generating a realistic sense of immersion, which is the technology 
objective aspect of VR (Gorman et al., 1999; Mestre, 2018). Optimal interpretation 
of the graphical images enables a fuller understanding of the virtual environment, 
and this experience is enhanced by high-resolution display devices viewed through 
a Head Mounted Display (HMD) (Burt, 1995; Coleman, Nduka, & Dazi, 1994; 
Gorman et al., 1999; Vince, 1995). To further enhance the sense of immersion, 
other sensory cues such as auditory and haptic can be added and there is consid- 
erable evidence demonstrating that more sophisticated VR technology results in 
increased presence (Banos et al., 2004; Diemer, Alpers, Peperkorn, Shiban, & 
Mihlberger, 2015; Gorman et al., 1999; Martin, Goncalves, Branco, Barbosa, & 
Melo, 2017). Another important element in the design of virtual environments is 
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interactivity because it allows the user to navigate, and can be achieved through 
hand tracking devices, motion coupled HMDs, and motion-tracking bodysuits 
(Burdea, 1996; Coyle & Thorson, 2001; Gorman et al., 1999; Jin, 2009; Keller & 
Block, 1997; Klein, 2003; Li et al., 2001; Steuer, 1992). The highly visual and 
interactive nature of VR has proven to be useful in understanding complex 3D 
structures and for training in visuospatial tasks (Hoffman & Vu, 1997). This 
recognition has meant there has been an increasing interest in developing VR-based 
applications for surgical education and training for over a decade (Gorman et al., 
1999), and according to Alaker et al. (2016), the incorporation of VR into surgical 
training curricula is now necessary. For surgical trainees, the development of 
low-cost VR surgery using more affordable headsets (e.g. Google Cardboard and 
Google Daydream) is vital (Riva & Wiederhold, 2015), given that desktop VR 
applications require expensive HMDs and high specification computers that are 
likely unaffordable for this target market (Pulijala, Ma, Pears, Peebles, & Ayoub, 
2018). Given that VR experiences are increasingly used for surgical training (Khor 
et al., 2016), a framework to build effective VR solutions is required (Pulijala et al., 
2018). Therefore, this study aims to determine the specific user requirements, 
usability, and design factors of VR for surgical training as a preliminary phase to 
developing a framework to build effective VR solutions in this specific context. 


3 Method 
3.1 Study Design 


A pilot study is a requisite step in exploring a novel intervention or an innovative 
application of an intervention (Leon, Davis, & Kraemer, 2011). Therefore, a pilot 
study was conducted to evaluate the feasibility of recruitment, assessment proce- 
dures, methods, and implementation of the novel intervention to be used in a 
subsequent larger scale study (Leon et al., 2011). In collaboration with Mativision, 
a VR content provider, a surgical video of a keyhole shoulder operation was 
recorded and developed into a VR mobile application. The viewing technology 
required for the intervention included a mobile phone inserted into a VR headset to 
view the surgical training video, and another mobile phone housed with the 
application Sensor Play to measure gyroscopic data. The mobile phone inserted into 
the headset was provided by each participant. 

To evaluate the effectiveness of VR for surgical training, this preliminary study 
employed a multimodal approach using three elements of data collection including 
measurements of head movement while experiencing VR, followed by a survey, 
and then a one-to-one interview. The interviews were conducted by an upper limb 
consultant in June 2018 and lasted between fifteen and twenty-five minutes. This 
included five minutes to watch the video and between ten and twenty minutes to 
complete the survey and for further discussion. The developed survey comprising 
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of 32 questions using Likert Scale (strongly disagree—strongly agree) was divided 
into three domains: (1) technical quality of the material, (2) usability and (3) user 
experience. Qualitative data was required to gather data from open-ended questions 
that could not be measured quantitatively from the survey. 


3.2. Sample and Participant Profile 


The population is Orthopaedic surgical trainees in the UK. From this population, 
nine participants from the North West were recruited at the end of their formal 
compulsory teaching. Participants were all males aged between 26 and 40 years old 
and represented a typical trainee orthopaedic demographic. Out of the nine par- 
ticipants, six were senior orthopaedic trainees and the remaining three where junior 
trainees committed to surgery, but not necessarily orthopaedic surgery. All par- 
ticipants were trailing the VR application for the first time. The VR application was 
appropriate for both levels of trainees because depending on their level of 
knowledge, each trainee will draw on a different skill from the VR application (e.g. 
focusing on the specific details of the task at hand, or gaining knowledge on the 
environment). To recruit participants, the upper limb consultant asked for nine 
volunteers to take part in the study at the end of the weekly teaching session. Nine 
trainees volunteered and took part in the data collection. 


3.3 Intervention 


For the intervention, each participant watched one five-minute 360° video on 
shoulder arthroscopy in VR. The shoulder arthroscopy video was filmed for the 
purpose of this study. In order to achieve this, one 360° camera was located near the 
primary operator for the duration of the surgery. Recording the surgery in 360° 
allowed the participant trainees to rotate their heads to explore the complete field of 
view. Two picture-in-picture screens were overlaid onto the video to provide more 
in-depth viewings and educational information. This included (1) a focused view of 
the operation and (2) a PowerPoint presentation with narration of the operation. 
Once the participant had trailed the VR surgery, both written and verbal feedback 
was obtained using the survey and interview, respectively. 


3.4 Data Collection 


During the qualitative and quantitative data collection, three main elements were 
explored. The first is technical quality of the material. This includes ensuring the 
intricate details of the 360° video are sufficient for surgical training, and whether the 
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video is viewable from a logistical and practical point of view. The second is 
usability, which explores the ease of use of the VR HMD, navigating the appli- 
cation, and learning from the 360° video and the picture-in-picture overlays. 
Finally, the user experience investigated the level of immersion achieved by the VR 
application, and its impact on educational outcomes. More specifically, a com- 
parison between VR and previous methods (e.g. watching videos) was made in 
order to draw out recommendations to improve the overall user experience. This 
was important because if the user experience is inadequate, then the longevity of 
VR for surgical training would not be maintained. Together with the qualitative and 
quantitative data, head movement data was measured. Gyroscopic data was 
recorded on a mobile phone, which was attached to the back of the participant’s 
head. 


3.5 Data Analysis 


A total score was calculated for each participant survey. From the initial survey 
score, it was evident that there was a consensus with the points that participants 
scored higher than others, which demonstrated areas that required further discussion 
and development, and those areas that were particularly effective. Therefore, the 
interview questions were used for the purpose of expanding on the survey outcomes 
only. To investigate the technical quality of the material, an example of the ques- 
tions asked is, “Was the quality of the video adequate for learning a procedure?”. 
An example of the usability questions include “How comfortable was wearing the 
headset?” , and for the user experience, “Can you tell me how the experience could 
be improved?”. The interview data was reviewed and a general consensus gathered 
in order to further support the discussion on the survey data. To analyse the 
gyroscopic data, the radian data files were exported from the mobile application to 
Excel, which demonstrated the calibration from each participant’s resting head 
movement. It was important to measure the head movement data because often 
surgical videos are recorded by attaching a camera to the light above the operation, 
and it is important for the light to be focused directly on the subject. With VR, this 
could mean that the user is looking down which could put a lot of pressure on the 
user’s neck and cause discomfort. Therefore, it was important to evaluate whether 
the angle on the VR video recording was appropriate for the user to have a com- 
fortable neck position while using VR. 
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4 Results 


4.1 Interview and Survey Data 


4.1.1 Technical Quality of Material 


The findings demonstrate that the quality of the VR 360° video was compatible and 
worked well with their mobile phones. The detail in the 360° video was clear 
enough to provide a sufficient view of the operating theatre (“{The video] provided 
a good overview of the whole operating theatre’, and, “It was good to see the 
arthroscopic view and which portals were being used at the same time, which other 
videos lack’). However, when displaying the intricate steps of the operations, 
improved graphics and video quality is required (“[The detail view] was very grainy 
and I couldn’t make out much”, and, “I think the video quality was worse than 
Vumedi’). The picture-in-picture video of the surgical field improved the detail of 
the important steps, however, this was subject to the resolution of the mobile phone 
used. The tutorial that was recorded using PowerPoint and displayed as 
picture-in-picture was perceived as useful in helping the participants orientate to the 
video and navigate the software (“7 wouldn’t have managed without the training 
video at the start’). Several participants felt that the position of the picture-in- 
picture could have been improved largely because it led to them looking down 
towards the ground which may become uncomfortable for prolonged periods 
(“[To improve the experience] made sure the arthroscopy view is easy to see...not 
too high’). Additional suggestions with regards to content include a “clear nar- 
ration of what’s happening” and “3D view of the shoulder’. 


4.1.2 Usability 


VR is advantageous given that it can be accessed remotely and from any location, 
which is attractive for this particular group of participants as expressed in the 
interviews. However, this is reliant upon the trainee having the appropriate 
equipment to view the VR 360° videos including a VR headset and mobile phone 
housing the surgical training application. Participants reported an average of a 20- 
30% drop in battery capacity during the viewing of the tutorial and a 5 min video. 
Hence, the usability limitations include having access to a VR headset, which may 
be centralised, and the reported drop in phone battery capacity after only 
five-minutes usage (“Battery life [would stop me using VR]. It killed my phone’). 
However, this could be overcome by providing trainees with a VR headset and 
mobile phone, rather than relying on participants using their own. Therefore, a 
recommendation to improve the experience was to provide a phone charger or the 
ability to charge the phone while using VR. An additional usability barrier is the 
limited number of videos readily available in VR at the moment (“having the [VR] 
goggles” and “not enough videos”). The weight of the VR headset was considered 
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suitable (“Not as heavy as it looks”) and several participants felt comfortable 
wearing the headset once they had gravitated towards wearing it (“J felt a bit 
disorientated at first...I forgot I was wearing it after the video started”). However, 
others suggested that the headset could be made more comfortable for those 
wearing glasses, which indicates the need for an improved focusing function (“It 
pushed my glasses into my face which wasn’t great’). 


4.1.3 User Experience 


The VR application was well received from all participants, however, recommen- 
dations to improve the user experience were provided. The system usability score 
(SUS) was 47, placing this technology in the lower 10th percentile rank of systems. 
This was largely due to a rather difficult user interface, which was evident given that 
participants reported issues navigating within the app to find the appropriate video, 
along with difficulties controlling the videos (e.g. stop/play/rewind) (“/The VR 
application] is not very intuitive...it needs a rewind button’, and, “I would like to 
jump forward to skip parts [of the video]”’). Five-minutes was considered a rea- 
sonable amount of time for VR surgical training videos. It was suggested that 
videos exceeding fifteen minutes would be too long to be used frequently, and 
therefore could reduce usage and acceptance (“J wouldn't wear it for more than 
15 min’). In addition, VR was considered more enjoyable than watching videos 
(“[VR is] much more fun than a normal video”), and allowed participants to have 
more control over the area of the video they were focusing on (“Jt was up to me 
what I looked at, not the cameraman’’, and, “I liked being able to look around and 
focus on more than just the arthroscopic view”). This is largely because of the 
immersive nature of VR in that participants felt like they were present in the 
operating theatre (“J felt part of the operation’’). Participants also felt that a more 
detailed picture-in-picture with waypoints to jump to certain parts of the operation 
would help with the learning, rather than treating it as a linear video. As a revision 
tool before any practical procedure the junior may undertake, it was clear that this 
360° video could be effective in providing a more detailed insight along with a clear 
explanation with the parallel PowerPoint presentation, when compared with a 2D 
video. This rich provision of multimodal information was widely accepted as a 
benefit over standard 2D videos. 


4.2 Head Movement Data 


Regarding head movement data, objective head movements showed that 88% of the 
time the head was in a neutral position or was looking slightly up (10-30°) towards 
the detailed view. Extremes of movement were generally avoided and when 
interviewed, participants felt comfortable and were not physically fatigued after 
viewing. 
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5 Discussion and Conclusion 


The aim of this study was to investigate the effectiveness of VR for surgical training 
and identify the user requirements for future VR applications in the field. Overall, 
this study demonstrated the importance of high quality resolution displays when 
developing VR surgical training applications, particularly when displaying the 
intricate steps of the operation. Although this is preliminary research, the findings 
indicated the effectiveness of VR as a training tool for surgical trainees compared 
with previous methods. More specifically, the design of the VR application used for 
this study provides trainees with detailed imagery as well as rich information, 
allowing them to apply theory to practice during remote virtual training. 
A limitation with regards to the usability is the trainee surgeons having access to 
VR hardware. Several improvements were suggested to develop user-friendly VR 
surgical training applications. This study suggests that VR could improve educa- 
tional outcomes for surgical trainees, although further research into the effective- 
ness of VR as a means of training is required. 


6 Theoretical Contribution and Practical Implications 


This preliminary study demonstrates several benefits of VR for surgical training 
with specific focus on trainee orthopaedic surgeons. The findings could be useful 
for VR developers in designing and developing effective VR surgical training and 
medical applications. Traditional teaching methods have been effective, however, 
given that emergent technologies such as VR provide important benefits in com- 
parison to traditional methods, it is important that health practitioners take advan- 
tage of such technologies. The ideal future scenario for a surgeon to train would be 
a complete VR simulation using near to real human tissue, without causing any 
patient harm. The use of immersive technology such as VR for learning in 
healthcare has scope for further progression to improve surgeon education and 
familiarity with a procedure before the trainee operates on a patient. A drawback of 
VR training is the cost involved (Bashir, 2010). In particular, substantial time and 
costs are required for VR simulator training (Yiannakopoulou et al., 2015), there- 
fore, this will likely be applicable for similar VR applications developed for this 
study. 


7 Limitations and Future Research 


A limitation of this study is the recruitment of a small sample of trainees at two 
levels of training and from one training institute within the UK. Residents, novice 
surgeons, and surgeons at various levels of expertise are required to learn basic 
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skills, to be trained in basic and advanced laparoscopic procedures or to be trained 
in rarely encountered situations, therefore, the effectiveness of VR training should 
be evidenced for different levels of human behaviour (Yiannakopoulou et al., 
2015). Another limitation of this study is that the gyroscopic data did not provide 
data on eye-tracking. Indeed, eye-tracking is important for future VR developments 
in the medical field because the unique feature of VR in comparison to 2D videos is 
that trainees can have a full immersive view of the operating theatre in 360°. 
Therefore, evaluating eye-tracking is important to determine whether trainees, 
particularly junior trainees, are taking advantage of the 360° element and viewing 
the surrounding environment. Although this study aimed to evaluate the educational 
impact of immersive learning, the data was limited. There is no standardised format 
to assess whether VR has a positive effect on surgical education, and future research 
is required to assess the educational impact of this technology on surgical training. 
For instance, future studies could evaluate whether immersive VR learning expe- 
riences results in higher retention for trainees in comparison to previous methods. 
This preliminary study provides a basis on which future studies will aim to develop 
a new methodology for evaluating VR training. 


References 


Akhtar, K. S. N., Chen, A., Standfield, N. J., & Gupte, C. M. (2014). The role of simulation in 
developing surgical skills. Current Reviews in Musculoskeletal Medicine, 7, 155-160. 

Alaker, M., Wynn, G. R., & Arulampalam, T. (2016). Virtual reality training in laparoscopic 
surgery: A systematic review & meta-anaysis. International Journal of Surgery, 85-94. 

Banos, R. M., Botella, C., Alcaftiiz, M., Liano, V., Guerreo, B., & Rey, B. (2004). Immersion and 
emotion: Their impact on the sense of presence. Cyberpsychology Behaviour, 7, 734-741. 

Bashir, G. (2010). Technology and medicine: The evolution of virtual reality simulation in 
laparoscopic training. Medical Teacher, 32(7), 558-561. 

Buckley, C. E., Nugent, E., Ryan, D., & Neary, P. C. (2012). Virtual reality—A new era in 
surgical training. In Eichenberg, C. (Ed.), Virtual reality in psychological, medical and 
pedagogical applications. IntechOpen (pp. 140-166). 

Burdea, G. (1996). Force and touch feedback for virtual reality. New York, NY: Wiley. 

Burt, D. E. (1995). Virtual reality in anaesthesia. British Journal of Anaesthesia, 75, 472-480. 

Coleman, J., Nduka, C. C., & Darzi, A. (1994). Virtual reality and laparoscopic surgery. British 
Journal of Surgery, 81, 1709-1711. 

Coyle, J., & Thorson, E. (2001). The effects of progressive levels of interactivity and vividness in 
web marketing sites. Journal of Advertising, 15, 65-77. 

Diemer, J., Alpers, G. W., Peperkorn, H. M., Shiban, Y., & Miihlberger, A. (2015). The impact of 
perception on presence and emotional reactions: A review of research in virtual reality. 
Frontiers in Psychology, 6, 26. 

Figert, P. I., Park, A. E., Witzke, D. B., & Schwartz, R. W. (2001). Transfer of training in 
acquiring laparoscopic skills. Journal of the American College of Surgeons, 193, 533-537. 
Gallagher, A. G., McClure, N., McGuigan, J., Crothers, I., & Browning, K. (1999). Virtual reality 
training in laparoscopic surgery: A preliminary assessment of minimally invasive surgical 

training virtual training (MIST VR). Endoscopy, 31, 310-313. 


316 R. Davies et al. 


Gallagher, A. G., Ritchie, K., McClure, N., & McGuigan, J. (2001). Objective psychomotor skills 
assessment of experienced, junior, and novice laparoscopists with virtual reality. World 
Journal of Surgery, 25(11), 1478-1483. 

Gorman, P. J., Meier, A. H., & Krummel, T. M. (1999). Simulation and virtual reality in surgical 
education. The Archives of Surgery, 134. 

Gurusamy, K. S., Aggarwal, R., Palanivelu, L., & Davidson, B. R. (2009). Virtual reality training 
for surgical trainees in lapaoscopic surgery. Cochrane Database System Review, 1. 

Hoffman, H., & Vu, D. (1997). Virtual reality: Teaching tool of the twenty-first century? Academic 
Medicine, 72, 1076-1081. 

Huber, T., Paschold, M., Hansen, C., Wunderling, T., Lang, H., & Kneist, W. (2017). New 
dimensions in surgical training: Immersive virtual reality laparoscopic simulation exhilarates 
surgical staff. Surgical Endoscopy, 31, 4472-4477. 

Jin, S. A. A. (2009). The roles of modality richness and involvement in shopping behaviour in 3D 
virtual stores. Journal of Interactive Marketing, 23, 234-246. 

Johns, B. D. (2014). The creation and validation of an augmented reality orthopaedic drilling 
simulator for surgical training (Ph.D. thesis). University of Iowa. 

Keller, P. A., & Block, L. G. (1997). Vividness effects: A resource-matching perspective. Journal 
of Consumer Research, 24, 295-304. 

Khor, W. S., Baker, B., Amin, K., Chan, A., Patel, K., & Wong, J. (2016). Augmented and virtual 
reality in surgery—The digital surgical environment: Applications, limitations and legal 
pitfalls. Annals of Translational Medicine, 4, 454-464. 

Klein, L. R. (2003). Creating virtual product experience: The role of telepresence. Journal of 
Interactive Marketing, 17(1), 41-55. 

Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role of interpretation of pilot studies in 
clinical research. Journal of Psychiatric Research, 45(5), 626-629. 

Li, H., Daughert, T., & Biocca, F. (2001). Impact of 3-D advertising on product knowledge, brand 
attitude, and purchase intention: The mediating role of presence. Journal of Advertising, 31(3), 
43-57. 

Martins, J., Gongalves, R., Branco, F., Barbosa, L., & Melo, M. (2017). Multisensory virtual 
experience model for thematic tourism: A port wine tourism application proposal. Journal of 
Destination Marketing & Management, 6, 103-109. 

Mestre, D. R. (2018). Immersion and presence. 1-10. 

Miiller, W. K., Ziegler, R., Bauer, A., & Soldner, E. H. (1995). Virtual reality in surgical 
arthroscopic training. Introduction to Journal of Image Guided Surgery, 1(5), 288-294. 

Nagendran, M., Gurusamy, K. S., Aggarwal, R., Loizidou, M., & Davidson, B. R. (2013). Virtual 
reality training for surgical trainees in laparoscopic surgery. Cochrane Database of Systematic 
Reviews, 8. 

Patel, H. R. H., & Patel, B. P. (2012). Virtual reality surgical simulation in training. Expert Review 
of Anticancer Therapy, 12(4), 417+. 

Pulijala, Y., Ma, M., Pears, M., Peebles, D., & Ayoub, A. (2018). An innnovative virtual reality 
training tool for orthognathic surgery. International Journal of Oral Maxillofacial Surgery, 47, 
1199-1205. 

Rambani, R., Ward, J., & Viant, W. (2014). Desktop-based computer-assisted orthopedic training 
system for spinal surgery. Journal of Surgical Education, 71, 805-809. 

Riva, G., & Wiederhold, B. K. (2015). The new dawn of virtual reality in healthcare: Medical 
simulation and experiential interface. Studies in Health Technology & Informatics, 219, 3-6. 

Scott, D. J., Bergen, P. C., Rege, R. V., Laycock, R., Tesfay, S. T., Valentine, R. J., ... Jones, D. 
B. (2000). Laparoscopic training on bench models: Better and more cost effective than 
operating room experience? Journal of the American College of Surgery, 191, 272-283. 

Seymour, N. E., Gallagher, A. G., Roman, S. A., O’Brien, M. K., Bansal, V. K., Andersen, D.K., 
et al. (2002). Virtual reality training improves operating room performance: Results of 
randomised, double-blinded study. Annals of Surgery, 236(4), 458-464. 

Steuer, J. S. (1992). Defining virtual reality: Dimensions determining telepresence. Journal of 
Communication, 42, 73-93. 


Evaluation of Virtual Reality in Orthopaedic Training ... 317 


Sutherland, C. J., Bresina, S. J., & Gayou, D. E. (1994). Use of general purpose mechanical 
computer assisted engineering software in orthopaedic surgical planning: Advantages and 
limitations. Computerised Medical Imaging and Graphics, 18, 435-442. 

Vince, J. (1995). Virtual reality systems. Reading, MA: Addison-Wesley Publishing Co. 

Yiannakopoulou, E., Nikiteas, N., Perrea, D., & Tsigris, C. (2015). Virtual reality simulators and 
training in laparoscopic surgery. International Journal of Surgery, 13, 60-64. 


Part VII 
VR and Media 


Towards the Essence of Cinematic VR: ®) 
Embracing New Technologies to Define crak 
a Medium 


Sarah Jones 


What is the essence of VR, its inner spirit, the cultural motor 
that propels the technology? 
(Michael Heim, The Metaphysics of Virtual Reality, 1994). 


Abstract Heim is concerned with the essence of Virtual Reality; the meaning 
behind the technology. These philosophical questions are not new. Heidegger, 
McLuhan and Heim are all influential in bringing together ideas about experience 
and technology that can be applied in a new way to immersive media. Coyne (1995) 
later brought the thinking together drawing on Heim’s idea that new realities and 
worlds would unfold within virtual reality. He urged us to take a Heideggerian 
approach that would value the technology in a new way, not bound or limited by 
previous constraints of what it is. As Heidegger explored the relationship with 
technology to be one that when experienced is able to freely develop to discover a 
true sense of meaning, McLuhan argued that the meaning can be found in the 
characteristics of the medium, not simply by the content (1964). This work is 
situated at the intersection between these philosophical approaches, media and 
technology to define a new genre of immersive media. It provides an introduction to 
the essence of immersive media, one where cinematic VR is no longer classed 
within film studies. Through an analysis of experiences that have been cited as 
defining the medium and pushing the boundaries of what is expected, new 
approaches are defined. Through a playful, experimentation approach, lessons in 
creating are documented to inform future forms of immersive experiences. This is 
no longer rooted in any one practice but, as Pimentel and Teixeira noted, merging 
all forms into VR, through simulation and stimulation, it’s the “emergence of a new 
holistic art experience of the mind” (1993:240). 


Keywords Cinematic VR - Media + Immersive technologies 


S. Jones (XJ) 
Birmingham School of Media, Birmingham City University, Birmingham, England, UK 
e-mail: Sarah.jones@bcu.ac.uk 


© Springer Nature Switzerland AG 2019 321 
M. C. tom Dieck and T. Jung (eds.), Augmented Reality and Virtual Reality, 
Progress in IS, https://doi.org/10.1007/978-3-030-06246-0_23 


322 S. Jones 


1 Introduction 


People come to the Oasis for all the things they can do. They stay for all the things they can 
be. Ready Player One (film) 


In the virtual world depicted by Spielberg in Ready Player One (2018), it is a 
world that allows for maximum opportunities and maximum personalities. The 
Oasis is virtual reality for those living in 2045. This is not a futuristic thought. 
Developments in immersive technologies allow for a range of interdisciplinary 
applications. Studies have shown the possibilities of responding to alien objects as 
if they are part of one’s body (Botvinick & Cohen, 1998) or believing that another 
body is one’s own (Petkova & Ehrsson, 2008; Slater, Spanlang, Sanchez-Vives, & 
Blanke, 2010). Other multi-sensory work has been used to demonstrate ultra-reality 
within art spaces (Ikei, Abe, Hirota, & Amemiya, 2012) and to develop work within 
memory studies (Dinh, Walker, Hodges, Song, & Kobayashi, 1999; Lehmann & 
Murray, 2005). The varied approaches have allowed virtual reality to permeate 
many aspects of life from training to education to entertainment and are spread 
across academic research. 

With the advancement in technology and accessibility placing considerable 
value on an industry estimated to be worth 40 billion dollars by 2020 (Orbis, 2017), 
it is time to review how we think about the technology. With a rush to flood the 
medium with applications and content, fundamental questions about what we learn, 
how we use and how the technology can transform thoughts and behaviours need to 
be addressed. These ideas have roots in philosophical traditions of Heideggerian 
thoughts (1975) and further developed by Heim asking “What is the essence of VR, 
its inner spirit, the cultural motor that propels the technology?” (1993). In later 
literature, Lanier noted that “the most important thing about technology is how it 
changes people (2010). 

In this chapter, these questions will be interrogated to challenge how we think 
about immersive technology to define a new medium. The approach is at the 
intersection of scholarly traditions in philosophy, media and technology and applied 
to recent practice. 


2 The Essence of Immersive Media 


The importance of defining immersive media begins with the provocation that 360° 
media and cinematic virtual reality is not situated within film studies, nor games 
studies. The call is to use the approach of philosophy, media and technology to 
create the future form, rather than developing an existing media practice. 

When we look at how media has developed throughout history, clear categories 
emerge from the first printing press (1440), the typewriter (1870) the telephone 
(1876), the radio (1895), the television (1925). It is a lot later that computer pro- 
gramme technologies were developed with the Collossus in 1943 and the microchip 
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in 1958. In 1960s, the first developments in simulations and virtual environments 
began to take place with Morgan Heilig and the Sensoramna in 1960 and Ivan 
Sutherland building the first headset, harnessed to the ceiling, in 1968. Augmented 
reality with virtual 3D objects imposed on real ones started being developed in 
1990 with the first augmented reality theatre production by Julie Martin in 1994. 
Despite this and Nintendo releasing the Virtual Boy in 1995, the technology pla- 
teaued and it didn’t become publicly accessible until the current wave emerging in 
2014. With this distinct timeframe for the technology emerging, it is clear how it 
forms a space in itself, rather than being aligned simply with film or television. 
Immersive media is distinct. It is not part of the gaming culture. Neither is it part of 
film culture. 

Literature in the field of immersive media is limited. Since the current wave of 
the technology emerged with the acquisition of Oculus by Facebook in 2014 and 
the accessibility of cardboard headsets in the same year, there has been a rise in 
research in the field of study but it is still limited. Traditionally scholarly work has 
focused on the areas of computer science (Burdea Grigore & Coiffet, 1994), edu- 
cation (Byrne & Furness, 1994) and health (Haluck & Krummel, 2000) but a new 
field of study is emerging (Jones, 2017; Mateer, 2017). 

It is critical to extend ideas and research to early stages of the emergence of the 
technology, particularly in the eighties and the nineties. It is at this time that 
significant research studies were being carried out and the conceptualisation of 
important notions. Presence is one of the defining features of immersive media and 
one that needs to be drawn out and understood. The notion of presence, which is a 
term that has multiple, contested scholarly origins. However, in the context of these 
varying definitions, the concept of presence discussed here is informed by Pimentel 
and Teixeira (1993), who describe the suspension of all disbelief in the world and 
believing that the mediated world is, in fact, not mediated as the condition of being 
present (Lombard & Ditton, 1997). It draws upon film theory (Ferri, 2007) and the 
origin of its term in literary theory by Coleridge, connecting engagement and 
suspension of judgment. 

Place is also an important feature in capturing a sense of presence in an expe- 
rience. Slater, Pérez Marcos, Ehrsson, and Sanchez-Vives, (2009) argued for Place 
Illusion (PD as one of two illusions necessary for an experience to be conducive to 
gaining presence. He argues that PI is how the world is perceived, with the second 
necessary illusion being Plausibility, relating to what is perceived, suggesting that 
the scenario being depicted is actually happening. The illusions are binary and are 
either experienced or not, “you cannot partially get an illusion” (2009:11). With 
emerging technology, creating a sense of Place Illusion is achievable but the 
questions rely on Plausibility to believe that this is happening at that specific time. 
Place and Plausibility Hlusions are developed by Skarbez, Brooks, and Whitton, 
(2017) with the emphasis on virtual embodiment as bring critical to achieving 
presence with 90% of participants surveyed citing the importance of this. The 
findings developed notions from De La Pefia et al. (2010) where presence was cited 
as being the combination of Place, Plausibility and the Virtual Body. Although De 
La Pena’s work is situated within immersive journalism, this conflicts with the 
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majority of factual storytelling approaches within immersive media, where there is 
no virtual body (Jones, 2017). Often this is the distinction between cinematic VR 
with 360° spherical cameras and CGI where limitations in technology often restrict 
embodied presence. 

To develop the technology to its full potential, it is useful to understand argu- 
ments that immersive media, as a practice, should be developing a distinct form of 
media and this may not use Place Illusion to create presence. Should the virtual 
world experienced be one that is distinct, not like reality but one deep in our 
imagination? This forms the arguments provided by Lanier (2017) beginning with 
the early emergence of technology when he argued that VR was like a Cyberdelic 
Experience (Barlow, 1990). This wasn’t an attempt to link experience to psyche- 
delic ones but rather because “‘it is as challenging to describe to the uninitiated” (in 
Barlow, 1990). In more recent works, Lanier has expressed concern that the 
potential of VR is being diminished, asking “What happened to the dream of 
improvising reality? Shared lucid dreaming? I mean, what’s the point of just 
making a flashier type of movie or video game?” (2017:6). This approach is at 
conflict with the elements of presence required in Slater and De La Pena’s work. 
There is no sense of relatability or plausibility in the environment and yet what 
Lanier argues is a world beyond conceived reality. In developing this line and 
applying it to cinematic VR, the argument here is that presence can still be found in 
this world and disbelief suspended. It is when we reach this point that we begin to 
see new practices emerging. As Lanier concludes, “this thing we seek, it’s a way of 
being that isn’t tied just to our given circumstances in the world” (2017:3). 

The field of immersive media is not new. It has evolved through decades of 
thought and work around story, narrative and computer science. Through applying 
previous theoretical ideas that were limited to computer generated virtual envi- 
ronments, the work can be applied in a new context. Walser’s idea of the “space- 
maker’ (1991) and Laurel’s definitive work on Computers as Theatre (1991, 2013) 
provide ideas of how we can locate presence within immersive storytelling. Laurel 
suggests theatre is a powerful metaphor to create virtual worlds: “at the theatre we 
relax our psychic boundaries to become engaged with the action, feel empathy with 
the characters and struggle with the problems enacted on the stage” (Pimentel & 
Teixeira, 1993:156). 


3 Immersive Media at the Intersection of Philosophy, 
Media and Technology 


The questions behind the technology were raised by Heim in 1994, asking: 


What is the essence of VR, its inner spirit, the cultural motor that propels the technology? 
(Michael Heim, The Metaphysics of Virtual Reality, 1994). 


These philosophical questions are not new. The technological influence from a 
phenomenological perspective of experience and discovery guides the thinking on 
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what the crucial parameters of immersive media are. Phenomenology allows for the 
understanding of the impact of technology on experience and consciousness 
(Borgmann, 1987; Dreyfus, 1992; Heidegger, 1977). Heidegger, Macquarrie, and 
Robinson, (1962) began to explore humanity’s relationship with technology as 
something that is experienced when it is able to freely develop to discover a true 
sense of meaning, with Merleau-Ponty (2013) analysing where our philosophy is 
developed by the experience of bringing truth into being. This approach seeks to 
articulate the essence of technology, where much deeper perspectives of immersive 
media can be developed as the technology advances. Heidegger explains that the 
essence of technology is nothing technological, arguing for a return to the begin- 
ning, before technological advances (Dreyfus, 1992; Heidegger 1977). The tech- 
nology does not make sense in itself so we need to understand what the technology 
is revealing to us, or disclosing to us, to fully understand its meaning and place in 
society (Heidegger 1977:2). 

As Heidegger describes: 


Whoever builds a house or a ship or forges a sacrificial chalice reveals what is to be brought 
forth, according to the perspectives of the four modes of occasioning. This revealing 
gathers together in advance the aspect and the matter of ship or house, with a view to the 
finished thing envisioned as completed, and from this gathering determines the manner of 
its construction. Thus what is decisive in techne does not lie at all in making and manip- 
ulating nor in the using of means, but rather in the aforementioned revealing. It is as 
revealing, and not as manufacturing, that techne is a bringing-forth 


In this approach, these ideas are applied to the practice of storytelling, and how 
new forms of technology permit new story forms to emerge, along with new 
patterns of experiencing the world and the stories it has to tell. A simple example 
may be found in a recent virtual reality piece by Marshmallow Laser Feast, which it 
titled “Treehugger’ (2016). In this work, the immersive experience asks the viewer 
to step inside the trunk of a tree and to experience its anatomy from within, which 
makes manifest a completely different way of viewing the object of tree. The 
experience from within is no less real or truthful from the view we have when 
looking upon a tree and, indeed, it may add an additional layer of reality to our 
comprehension of what a tree entails. 

This example helps to realise more recent philosophical debates within the field. 
Debates around consciousness have asked questions concerning the meaning of 
virtual reality and how we think about our relationship to it. From the idea of 
cyberspace acting as a virtual world to understand cultures, illuminating “the 
magical reality of all human narratives” (Pesce, 1997:12) to questioning the 
provocation itself, asking ‘why do we feel the need to create something when we 
seem to have so little understanding of why the natural exists?’ (Gigliotti, 1997:40). 
Chalmers has furthered the debate about the reality of virtual reality, arguing that it 
is in fact a form of reality itself (2017). Attributing the value to life in virtual world 
as almost equivalent to that in non-virtual worlds, this line of argument creates a 
foundation in which we can place value on immersive media to create genuine 
experiences that hold value and create knowledge. 
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The philosophical ideas can be drawn out further through scholarly work in 
media and technology, particularly in McLuhan’s writing on media and his argu- 
ment that the “medium is the message” (1964). It is deliberately a paradoxical 
statement but one where McLuhan argues that when analysing a medium we 
shouldn’t look at its content but at the inherent characteristics of a medium itself. To 
draw out the argument, McLuhan cites the example of a lightbulb; it doesn’t have 
the same level of content that a newspaper or television has but still it is a medium 
that has a social effect, as “a light bulb creates an environment by its mere presence” 
(1964:8). For McLuhan, the focus must be on the form that communication is being 
delivered in, what is being communicated is less important. 

So throughout history, what has been communicated is less important than the 
particular form that the communication has been delivered in. The emphasis on the 
visual within the printed medium, or oral cultures within speech mediums and 
television and radio unifying people to encourage participation (Heydon, 2017; 
Levinson, 2003). With this approach, it’s the technology that transfers the message 
that changes us and changes society. This is the idea that Lanier continues to come 
back to, arguing in 2010 that the most important thing about a technology is how it 
changes you (Lanier 2010:4). 

These perspectives allow us to think differently about immersive media, ques- 
tioning what it is that reveals itself through the medium, drawing on both 
Heideggerian thought and McLuhan. Heim argued that there were seven elements 
that constitute the idea and essence of virtual reality (1993:108-127); 


1. Simulation refers to the modern computer graphics and sound effects that can 
create such a high degree of realism. 

2. Interaction refers to electronic representations that people perceive as virtual 

reality by their interacts with them. 

. Artificiality refers to a world that is largely of human construct. 

4. Immersion refers to the computer-generated sensations to which a man can 
immerse his sensory perceptions to simulate reality. 

5. Telepresence refers to the capacity of computer technology to replace human 
presence by robotic presence. 

6. Full-body immersion refers to the latest technology that allows human body to 
interact with graphics on a computer screen. 

7. Networked communications refers to communication that can go beyond verbal 
or body language to take on magical, alchemical properties. 


1S) 


Situating Heideggerian thought within the emerging technological form, Heim 
argues that if an experience has these seven constituents, it will lead to a philo- 
sophical experience (1993:36). It strengthens the argument that the focus should be 
on the revealing of the technology, rather than the technology itself. 

Placing the value within the technology when it is not bound or limited by 
previous constraints of what it is forms Coyne’s arguments (1995). Through this, 
new realities and worlds unfold within virtual reality. Coyne argues that the taking a 
different approach can have merit. In appreciating that claims that using VR reveals 
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something new about the world doesn’t work if we think about truth as disclosure. 
Nor does it work if we think solely about perception. However, we can find new 
meanings if we look at it through different metaphors. As he argues, 


In appreciating that using VR is not like operating puppets, we see that we are not con- 
strained (as though by strings), that we can achieve something other than entertainment, 
that the puppets can change identity to become the people operating them. The VR 
experience is not like walking through a building—we can fly through it, pass through 
walls, and shrink and expand the building around us (1994:71) 


Through these approaches of philosophy, media and technology and look at 
them through immersive media, the discovery is the essence of virtual reality, 
rooted fundamentally in presence, leading to a philosophical experience. 


4 Immersive Media in Practice 


A number of experiences have been analysed to apply the above concept of 
immersive media. The experiences based on peer-review and curated collections at 
Tribeca and Sheffield DocFest Alternate Realities. These run in April and June 
respectively and for this study offered exploration of work that is seen as defining 
and the best in the field of study. The Alternate Realities section at Sheffield 
DocFest (2018) showed 27 interactive and immersive projects including films, 
games and web projects. At Tribeca (2018), immersive, interactive and emerging 
technological works are situated within the StoryScapes and Tribeca Immersive 
strands. 25 projects were shown. The work is selected by a jury, curated by leading 
professionals in the immersive media. It is designed to show the best work at this 
present time. Four projects have been selected for analysis, based on the fact that 
they were shown at both festivals. This study draws on techniques and audience 
engagement with the experiences to gain an understanding of how the industry is 
progressing and where new techniques and methods are being realised. 


Terminal 3 


PROJECT CREATOR: Asad J. Malik 
COUNTRY: USA Pakistan 

YEAR: 2018 

DURATION: 12 min 


The Day the World Changed (https://sheffdocfest.com/films/6564) 


PROJECT CREATORS: Gabo Arora, Saschka Unseld 
COUNTRY: USA Japan 

YEAR: 2018 

DURATION: 10 min 
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Vestige (https://sheffdocfest.com/films/6566) 


PROJECT CREATORS: Aaron Bradbury, Paul Mowbray 
COUNTRY: UK France USA 

YEAR: 2018 

DURATION: 15 min 


This is Climate Change: Feast and Famine (https://sheffdocfest.com/films/6585) 


PROJECT CREATOR(S): Danfung Dennis, Eric Strauss 
COUNTRY: USA Somalia Brazil Greenland 

YEAR: 2018 

DURATION: 18 min 


Empathy-driven media has dominated cinematic VR in the past few years 
(Jones, 2017) with the idea that VR is to show how one can ‘walk in someone else’s 
shoes’. As Mandy Rose stated in an iDocs overview of Sheffield, “I wish we could 
get past this preoccupation with walking in the shoes of another, seeing through the 
eyes of another” (iDocs, 2018). This can be aligned to Coyne’s thinking about not 
replicating a world but looking at it through a new lens, something that Lanier has 
developed insisting that virtual reality should be creating different worlds and 
different realities not replicating what we have (2017). 

Terminal 3 is an interactive, augmented-reality documentary, using holograms 
and personalised experiences to drive forward the experience. Six participants of 
Muslim descent were filmed with Depthkit technology. In the experience, the 
participant takes on the role of a customs office and interrogates the suspects. The 
experience lasts for fifteen minutes with the questions for the interrogation 
appearing within the field of vision. The voice triggers responses. Fernando 
(Guardian 2018) tried the experience at Sheffield DocFest; 


It puts me in the position of authority—do I choose the tough questions or the softer ones? 
At the end I’m asked to decide whether to let her into the country or detain her further. 


The experience uses the Hololens, Microsoft’s augmented reality (AR) glasses. 
The fictional airport is created around you and the encounters become more per- 
sonal as the experience develops. What is interesting, in an artistic statement of the 
work, is the photorealism of the person that you are interrogating. They initially 
appear like a ‘digital ghost’ and then depending on the questions that you ask, they 
can gradually appear more lifelike (Ha in TechCrunch 2018). 

The creator, Malik, maintains there is no specific message that is being promoted 
but instead it is about illustrating the enormous variety of personalities, back- 
grounds and viewpoints among people who may or may not identify themselves as 
Muslims, but “who the world would identify as Muslims”. This develops the work 
on Shameful Conquest (2017), a post-Brexit experience, where no political message 
is being articulated, it’s just an experience of life post-Brexit. 

The difference with this project comes in the utilisation of AR as a means of 
storytelling and this extends the practice of immersive media into a new genre. With 
augmented reality placing digital objects in your environment, the interaction takes 
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you one step further, “Terminal 3’s power comes from forcing you to share your 
physical space with the holograms” (Schwab, 2018, CoDesign) 

The responses online from the audience signified the transformative nature that 
this type of experience had: 


Kudos to @ AsadJMalik (https://twitter.com/AsadJMalik) Terminal 3 was one of the most 
subtle on the surface and profound upon analysis experiences at the Sheffield Doc/Fest 
Alternative Realities Exhibition this year, #SheffDocFest (https://twitter.com/hashtag/ 
SheffDocFest?src=hash) 


David Tames (@cinemakinoeye) 


I’ve never cried in a VR/AR experience, but I had the honor of going through #Terminal3 
(https://twitter.com/hashtag/Terminal3?src=hash) at #Tribeca2018  (https://twitter.com/ 
hashtag/Tribeca2018?src=hash) and I cried for a solid 2 minutes at the end. So, so pow- 
erful. And as I’m reflecting, writing about my experience for a class—I find myself crying 
again as I tell the story. 


Such a powerful, powerful experience. Using voice was brilliant, to put people in such an 
uncomfortable position. It makes me realize how easy it is to disassociate from your body, 
clicking buttons on a controller—but how much harder it is to dissociate from your own 
voice. 

Rogue Fong (@__roguef) 
Incredible experience that leverages the power of AR, presence, and embodiment. 


Helen Situ. @helensitu 


The Day the World Changed was created in partnership with Nobel Media and 
Nobel Peace Prize Laureate and the International Campaign to Abolish Nuclear 
Weapons. It is one of many experiences that have been supported and developed by 
third-sector organisations, developing the arguments for empathy-driven content 
(Jones, 2017). The experience uses first-hand testimonies of the victims and sur- 
vivors of atomic bombs and nuclear arms testing. It brings this together with data 
visualisations and 3D scanning and photogrammetry. It is the most like the 
empathy driven content that previously dominated the field, unsurprising as the 
creator is Gabo Arora (Clouds Over Sidra 2015). Arora is known for his work with 
VRSE and the United Nations so is well versed on developing immersive media that 
helps to change understandings; 


This experience uses virtual reality’s strengths in a way that makes viewers have a more 
visceral relationship with what is usually just a rational engagement with numbers, this time 
focusing on one of the most urgent issues of today 


Arora (2018) 


Although agency is limited in the experience and it follows a directed narrative, 
there is an element of the VR experience infiltrating the environment around the 
participant. Within the space, the participant can interact with objects recovered 
from the site. This looks at the development of situating experiences in an envi- 
ronment to deepen the sense of presence, something that has been a focus of 
literature within immersive theatre (Alston, 2013; Laurel, 2013; Zaunbrecher, 
2011). These studies have been influential in the development of exhibiting work 
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but also through increasing presence through environment. In response to the work 
at Shef Doc Fest, Julie Lennox noted the increase in utilising the environment to 
enhance the experience and asked, “is this perhaps some indication that creators are 
realising we need something more than the virtual experience?” (Stevenson, 2018). 

The use of capturing testimonies from survivors is interesting and raises further 
research questions about the role that immersive media could play in capturing 
memories and digital archives or places and environments, which is an area for 
further research (Koller, Frischer, & Humphreys, 2009). 

The audience responses contained less positive terms than The Terminal and 
were also limited in number. However the aims and objectives of the experience do 
appear to be met in some of the engagement. Saschka Unseld, the project’s 
co-creator, had stated on the project’s website, “we want this to be an unwavering, 
uncomfortable experience for people”. Feedback from participants said, 


#TheDayTheWorldChanged _ (https://twitter.com/hashtag/TheDayThe WorldChanged?src= 
hash) Disturbing experience, yet so real about Hiroshima and the consequences by 
@gaboarora (https://twitter.com/gaboarora) 


Daniela Baicoianu: @DaniiAlina 


Ideas being ahead of the technological limitations other reports suggested the 
experience was too chaotic in user design, complicating the message and hindering 
the opportunity for presence. These concerns have dominated literature with 
arguments in co-creation with participants when working toward user experience 
(Kohler, Fueller, Matzler, Stieger, & Filler, 2011) and utilising applications like 
teleportation to reduce motion sickness (Hong, 2017). Other researchers have 
produced frameworks based on focus group studies to understand the design pro- 
cess for immersive experiences, taking into account the ease of discovery, the visual 
input in the field of view, limiting gestures and hand movements and ensuring that 
the design gives the user a clear focus as too much multi-tasking can become 
overwhelming (Spillers & Mortensen, 2018). 

The user experience of The Day The World Changed seems at conflict with the 
last criteria for virtual reality design, according to participant testimonial. Julia Scott 
Stevenson, researcher at University of West of England found it to be; 


a great example of trying to do too much—both in terms of tools and experience. There 
were so many different elements that didn’t hang together very well: a lot of mismatched 
visual content, the interaction with some items that triggered audio was a bit clunky, and the 
blue dots coming out of our hands at the end were quite confusing as to what they were for 
and how much control we had over them. 


Scott-Stevenson (2018) 


The strength of this type of experience still lies in the idea of empathy and the 
sense of understanding how other people live. 


Experiencing together with many others #TheDayTheWorldChanged (https://twitter.com/ 
hashtag/TheDayTheWorldChanged?src=hash) by @NobelPrize  (https://twitter.com/ 
NobelPrize) @nuclearban (https://twitter.com/nuclearban) @gaboarora (https://twitter. 


Towards the Essence of Cinematic VR: Embracing New Technologies ... 331 


com/gaboarora) @saschkaunseld (https://twitter.com/saschkaunseld) & @igaln (https:// 
twitter.com/igaln) I achieved a sensation of doing something right. 


Captain Strange @ KapoStrano 


The words “doing something right” are typical in analysis of empathy-driven 
immersive experiences (Jones, 2017). However, the depth of reaction and discus- 
sion in online forums is not to the same extent as Terminal 3, which involves much 
of an active experience. There is much more excitement around the development of 
the technology and new ideas signifying a potential audience fatigue with this genre 
(Moeller, 2018). 

This is Climate Change is part of a series of experiences, produced by 
Participant Media and Condition One. They focus on climate change and looks at 
how human life and actions are destroying the planet. The first episode looked at 
glaciers in Greenland, moving to the Amazon rainforest, wildfires in California and 
famine in Somalia. It follows a traditional VR documentary style with the same 
techniques and film grammar that we are becoming familiar with (Mateer, 2017). 
Like Arora above, the intention of this docs-series was to feel the story and 
experience it, rather than watching as a detached observer (Hardarwar, 2018). 


We are getting to a place where VR is a stream of conscious experiences. We need to get 
out of the way as filmmakers to allow these raw experiences to be told. It is heartbreaking to 
see how climate is affecting people in #ThisIsClimateChange (https://twitter.com/hashtag/ 
ThisIsClimateChange?src=hash), 


@Danfung (https://twitter.com/Danfung) & #EricStrauss  (https://twitter.com/hashtag/ 
EricStrauss?src=hash) 


Again, an analysis of social content was limited. Participant’s experiencing This 
Is Climate Change may have had a transformative experience, but didn’t promote 
this in online forums. What is sourced is restrained, although full of praise. 


Just saw #thisisclimatechange (https://twitter.com/hashtag/thisisclimatechange?src=hash) at 
the Tribeca Film Festival. Awesome virtual experience. Check it out. #EarthDay (https:// 
twitter.com/hashtag/EarthDay ?src=hash) 


Tanya Morgan @MemphisMorgan 


Vestige is also a non-fiction experience, based on the personal experience of 
Lisa Elin, following the death of her husband Eric. It is similar in style and tone to 
Notes on Blindness (2017) taking the participant through her mind to witness 
memories. It takes a multi narrative approach, with memories coming together. 
Creators used volumetric capture of two actors who played Lisa and Eric. More 
than 15 h of audio interview was captured of Lisa remembering her husband and 
the times they spent together. Like Notes, it is the audio that drives the experience. 

Analysis of participants online all pointed to the emotional power of the piece 
and the difference in how the technology was used in a way that couldn’t have been 
told by any other medium. 
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Me experiencing one of the most emotional VR projects I’ve seen so far @sheffdocfest 
(https://twitter.com/sheffdocfest) Vestige is awesome and a must see! #vestigevr (https:// 
twitter.com/hashtag/vestigevr?src=hash) #sheffdocfest (https://twitter.com/hashtag/ 
sheffdocfest?src=hash) #vr (https://twitter.com/hashtag/vr?src=hash) 


Anne Doncaster: @IMMDesigns 


Vestige made me really cry, and the animation was beautiful, but at the end, I was left 
wondering whether the monologue could have been delivered in some other way. 


Scott-Stevenson (2018) 


From the analysis of participants responses, there is a clear correlation between 
those that are embracing the technology in new or different ways as gaining a 
deeper response. This could simply be down to the audience seeing something new, 
pointing to a need to continue to experiment and engage with different technologies 
to develop the medium. 

The experiences above are often being created through triggers of personal 
experiences. Although this is often the case of personal-focused documentaries, in 
other media practice, ideas are increasingly being developed through social media 
and crowdsourcing (Simula & Vuori, 2012; White, 2010). The triggers for both The 
Terminal and Vestige were personal experiences of the directors (Asad J. Malik, 
The Terminal) and the protagonist (Lisa in Vestige). 


5 Conclusion 


This chapter focuses on the intersection of immersive media between philosophy, 
media and technology, understanding how a distinct form of presence can be 
manifested. Through a methodology to analyse this approach through recent 
practice, it is clear that none of the selected works could be experienced in any other 
form, other than virtual reality. However, the two pieces of work that were more 
traditional documentary style and both empathy-driven (This Is Climate Change, 
The Day the World Changed), provoked less of a response and also used the 
technology in a more limited capacity. 

The focused experiences all bring elements together that demonstrate that 
intersection of philosophy, media and technology. If we don’t treat it or define the 
medium differently, we simply end up with ‘walking in someone else’s shoes’ 
filmic practice, that doesn’t always justify why it needs to be in VR. To develop the 
technology, the boundaries need to be explored to understand how it is different and 
how we can be changed by it. 

In early studies on virtual reality, Pimentel and Teixeira (1993) encouraged the use of 
VR as a form of art where artists, through simulation, will “merge together all art forms” 
(1993:240). Through doing this, it would become an art experience of the mind. It’s 
something drawn on in more recent literature by Ryan (2015) acknowledging that VR 
has the potential to be “total art”, in a way that no other medium has achieved. 
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It is Heim and Lanier that offer closing thoughts that have influenced this work 
and my future projects. With emergent media, the big picture can get lost, in what 
Heim describes as the rush for content to fill the new medium. It is down to artists to 
guard the visionary aspects and nurture it in its infancy (1995:66). It is a warning to 
“not lose itself in the thrill of the content’, the idea reiterated by Lanier on many 
occasions. A question that remains prominent at the end of this thesis, posited 
earlier, 


What happened to the dream of improvising reality? Shared lucid dreaming? I mean, what’s 
the point of just making a flashier type of movie or video game? (2017:6). 


Experiences 
Terminal 3 


PROJECT CREATOR: Asad J. Malik 
COUNTRY: USA Pakistan 

YEAR: 2018 

DURATION: 12 min 


(https://www.beggarkings.com/terminal3/) 
The Day the World Changed (https://sheffdocfest.com/films/6564) 


PROJECT CREATORS: Gabo Arora, Saschka Unseld 
COUNTRY: USA Japan 

YEAR: 2018 

DURATION: 10 min 


(https://www.thedaytheworldchanged.world) 
Vestige (https://sheffdocfest.com/films/6566) 


PROJECT CREATORS: Aaron Bradbury, Paul Mowbray 
COUNTRY: UK France USA 

YEAR: 2018 

DURATION: 15 min 


(http://vestige-vr.com) 
This is Climate Change: Feast and Famine (https://sheffdocfest.com/films/6585) 


PROJECT CREATOR(S): Danfung Dennis, Eric Strauss 
COUNTRY: USA Somalia Brazil Greenland 

YEAR: 2018 

DURATION: 18 min 


(https://www.beggarkings.com/terminal3/). 
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